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PREFACE 



\ 



Sir Edmund Hillary's climb to the summit of Mt Everest an 28 May, 1953, 
is widely acclaimed. What is much less acclaimed, however, is the important 
photographic record of the entire expedition. Not only to climb, but also 
to record the. process, is to perceive one's potential responsibility to 
future climbers whatever their summits. In educational innovation, our 
ability to create and implement new projects often outstrips- our ability 
^ ^ to re nder a dequate accounts of these innovations. As Glass _( 19721. uptejSUL. 

/' Innovative programs are prone to vanish without a trace ... 

Without a well-kept historical record, this year's innovation 
may be a repeat of last year's debacle. (Glass, 1972, p.. 105) 

There is a scarcity of examples of recent Australian school-level 

innovations which have been thoroughly documented. In fact, one of the 

few available 'sources of valuable and comprehensive documented' records of 

innovations are reports of the "augmented" evaluation (a process it^wh^ch 

an external consultant assists an innovator in collecting and reporting 

information) of numerous projects funded by the Schools Commission 

Innovations program. " ' i 

« 

The central purpose of this monograph is to provide practical 
assistance to those involved in school-level innovations (whether funded 
by the Innovations Program or not) in the important task of "evaluating 
and documenting their projects* , One of the noteworthy features of the 
monograph which should be of special interest to innovators is that it 
is base ; d almost, entirely on authentic apd concrete examples culled from 
reports of augmented evaluations funded by the Innovations Program.' 

The National Innovations Committee of the Schools • Commission 
Innovations Program commissioned the development of this monograph and 
approved and funded publication in its present form. The views expressed 
here, however, do not necessarily represent the views of the Schools 
Coranission* * 
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CHAPTER 1 : INTRODUCTION AND BACKGROUND 



The main, aim of this monograph is to give practical guidance to % 
innovators wishing to evaluate and document their own projects. The 
approach 'taken in later chapters rests heavily upon the use of concrete 
illustrations drawn from report's of evaluation of innovative projects 
funded under the auspices of the Schools Commission Innovations Program. 
In order to facilitate understanding of later chapters and to place them 
into context, discussion in this introductory chapter provides background 
information about (a) the Innovations Program, (b) evaluation approaches 
in the Innovations Program, (c) the place of evaluation in the innovative 0 
process and (d) the present Monograph's orientation,- 



INNOVATIONS PROGRAM 

The Innovations Prograjn - or what is now officially called the Special 
Projects Program (School-level Innovation) - began soon after publication 
of the Karmel Report (Interim Committee of Australian Schools Commission, 
1973); According to the Karmel Report, "the intention of the special 
projects grants is to encourage spirited^^hough responsible, collaborative 
efforts to improve the quality of schooling" (p. 128), The basic philosophy 
of the Innovations Program, as outlined in a chapter of the Karmel Report 
entitled "Fostering Change", is to improve the quality of Australian 
education by encouraging change or innovation at the "grass roots" level. 
That is, the Innovations Program is premised on the belief that educational 
problems should be identified by people at th^ school level (e.g., teachers, 
community groups, parents, students) and solved by these people in their . 
own terms. This model of change can be contrasted with the common "top down" 
model involving imposition of policies for change £rom above. The annual 
budget made available in different years to support school-level innovative 
projects ha^ ranged between two and three million dollars (approximately 
one per cent of the Schools Commission's total budget). 

To be eligible for Innovations Program funding, a project proposal 
needfe to satisfy several criteria (see Maxwell, 1978), First, although . * 
projects needn t provide completely new educational ide^as', s they must be 



innovative within their own contexts. Furthermore, because of this 

emphasis on innovativeness within context, it is not essential that a 

funded project be replicable in or transportable to other situations. 

Second, projects should offer reasonable likelihood of benefits (e.g., 

advantages for target groups, encouragement of further developments, 

long-term changes in educational practice). A third/ set of criteria 

concerns consistency of the proposed project with the priorities of the 

Schools Commission (e.g., increasing and advancing equality of educational 

opportunity, community involvement in schools, basic skills, diversity of 

choice, and cross-cultural and racial sensitivity). 

\ 

Some idea of the very considerable variety in the types of project^ 
funded by the InnovationsSProgram can be gleaned from the Schools 



Commission f s (1976, 1979) directories of projects. The 19 79 directory^ ^ 
classifies funded projects under the following 13 ma jor -headings : 
Teaching Methods, School Organisation (Inter and Intra), Curriculum Areas 
(e.g., drama and dance, mathematics, vocational education and trade, whole 
school curriculum development), Remediation in Literacy and Numeracy, 
Special Education - Handicapped Children, Migrant and Multicultural 
Education, Disadvantaged Groups, Sexism: Equality of Opportunity, Teacher 
Support (production o'f educational materials, teacher enrichment/development), 
Extra School Activities (camps* outdoor activities, field studies^ after-school 
programs, playgrounds, availability of school facilities), Community-School 
Relationships (students in the community, schtfol-community interaction,, 
parents and community contributing to school programs), School and Work, 
Learning Centres and Networks. 7 

The implementation of the Innovations Program depends in an important 
way on a group of^ part-time personnel called "counsellors' 1 . Theses counsellors f 
'activities include appraising applications (through 'face-to-f ace contact with 
applicants). and rank ordering of grants, providing stimulation to potential 
innovators, offering support for projects, and giving advice on how to 
evaluate projects. Further information about the philosophy and organisation 
of the Innovations program is contained in Schools Commission reports 
(e.g., Schools Commission, 1975a, 1978), several articles (e.g., Musgrave, 
19W; McKinnon, 1976; Porter, 1976; Pusey, 1976) and a recently assembled 
collection' of internal program documents (Schools Commission, 1980), 
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EVALUATION APPROACHES IN INNOVATIONS PROGRAM 

The Schools Commission (1975a, p. 198) expressed commitment to 

evaluation, but distinguished evaluation of th<T Innovations Program as 

a whole from ^valuation of individual proUcts funded under the Program. 

The only major report on Program evaluation "has been completed recently 

(Foyster, 1978), and this provides insight into Program operation and 

valuable statistics about the changing pa.tterns of grants to different 

states, school systems, types of projects, etc. Initially no special, 

provision was made for ' evaluation at the project level s&that, "when an 

* » * ) 

evaluation component was introduced into the program, it was grafted onto 

an already flourishing tree 11 (Maxwell, 1978, p. i). 

During 1975, a document entitled "Guidelines -for the Evaluation of ^ 

Innovations Projects" appeared -<Sefroat^Gomnris«si on, 1975b) ,— and this 

afforded an important place to evaluation: U A1°1 school-level projects 
which receive grants under the Commission's Innovations Program will be 
evaluated" (p.l)* Also, in 1975, a small group of Program counsellors in 
each State and Territory was convened to facilitate implementation of these 
evaluation guidelite*£,. \- * 

The-1975 guidelines delineated two distinct types of evaluation, 
called general evaluation and augmented evaluation . General evaluation 
was to be undertaken by project directors themselves ,and reported as part 
of the final project report '(with the possibility of obtaining advice 
from the small group of counsellors looking after evaluation%in each 
State and Territory or of toeing placed in contact with other innovators 
working in similar fields). On the dther hand, augmented evaluation 
involved the appointment and funding of an external evaluation consultao* 
to assist ^the project director in planning, conducting and reporting an 
evaluation more # comprehensive attff substantial than general evaluations 
normally would be. Aj^ important feature of augmented* evaluations is that 
they were conducted only if the project director approved of the evaluation 
plan and the particular evaluation consultant. This is consistent with the 
Program's philosophy that project directors should at all times remain in 
control of all aspects of their projects including evaluation. Important, 
merits in involving external evaluation consultants are that the^ had time 
and funds set aside for evaluation, they had expertise and experience in 
evaluation, and they were sufficiently' remote from day-to-day operations 
to take an independent view of projects. 

C 11 



As 'part of the.'process" of interpreting and implementing the 1975 
evaluation guidelines, the group of Program counsellors specifically 
involved in evaluation in N.S 0 W. produced a useful working document 
entitled n Augmented Evaluation: Purposes and Procedures" (Straton, 1977X* 
Although this document referred especially to augmented evaluation 
procedures followed in N.S.W. \ it is likely that fairly *simi lar apprbaches 
were implemented 1 in otffer States and Territories. The first step 
delineated <in Straton's document was the selection of a project for 
augmented evaluation. A second step involved obtaining the, project 
director's approval for the augmented evaluation, The third stage 
involved the nomination of an external evaluation consultant and, if 
both the project director and the consultant were agreeable, the 
preparation of a costed plan for'tfie ^valuation. A fourth step involved 



Program counsellors invetting evaluation-plans to check their suitability 

and acceptability to the project director The fifth phase involved the 

conducting and reporting of the evaluation by the consultant in clpse 

liaison with .the projectTHirector and the evaluation group. „ 
■ 

Mainly during 1976 and 1977, the groups of counsellors involved in 
evaluation in Various States And Territories initiated the augmented 
evaluation of approximately 80 Innovations Program projects which had 
been granted funding in the period 1974-1976. Although other similar 
evaluations (later referred to as Commission-initiated evaluations) were 
conducted in 1978 and subse^|nt years for projects in Series Q and 
subsequent series, the brierfor the present monograph involved examination 
only of augmented evaluation reports for projects prior to Series Q. 



PLACE OF EVALUATION IN INNOVATIVE PROCESS 

There is a divergence of opinion about the place and importance of 
evaluation in any plan for innovation. It has been noted in an Innovations 
Program document that M the Commission is conscious that innpvative projects 
can easily be inhibited or distorted by the demands of evaluation" 
(Schools Commission, 1975b, p. 5). Maxwell (1978) has summarised the 
following objections* to evaluation which have been raised by various 
people associated with the Innovations Program: ^ 
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Evaluation may stifle innovation rather than encourage it; to 
subject projects and project directors to critical scrutiny may 
> encourage a retreat .tq "safe" ideas, may discourage risky' ventures,* 
■ and may even lead to fewer proposals. Teachers and others,' unused 
to much responsibility, for identifying and solving local educational 
problems may. need a lot of support, and encouragement even to 
formulate a proposal. Evaluation may undercut the growing con- 
fidences of such people in their own capacities." Priority needs to 
be placed on fostering this confidence , on generating a clima£e ' . 
within which educational innovation can occur and be sustained! 
_In the experience of most people, evaluation has been moj^ly 
judgmental,, negative and competitive. Why, then, place oneself in a 
situation where one's best effort's, one's best ideas, are looked at 
critically by an outsider who may very* likely create considerable 
discomfort and insecurity? (adapted from Maxwel 1 , *1978', p. 6) 

* » ■ 

Although Maxwell believes' that most of these statements reflect naivety 
about evaluation, if is important to note that these feelings about 
evaluation did exist. 

In contrast to the above, Goldberg (1971) and- Nisbet (1974) propose 
cogent arguments supporting the central importance of evaluation in the 
innovative process. Because innovation has become' something of a bandwagon, 
Nisbet warns anyone who plans to jump on the bandwagon of innovation* to 
consider its .potentialities as a hearse. " F or anyone who embarks on any 
innovatory experiment in education encounter's a number of problems which 
may prove disastrous not only to his experiment but Aso to himself and 
to his^colleagues in the experiment" (p. 5). These problems include an 
increase in everyone's work ltfad„ a* loss 'in confidence and an increase in 
anxiety (since the effect of innovation is to destroy the teacher's 
competence), confusion which inevitably accompanies novelty, and backlash 
(e.g., blaming innovators for educational ills) 

r . - 

Nisbet considers evaluation an essential requirement if we are to 
,have innovation in schools, and makes the, following claim; 

v 

, In education we are at the stage of the Wright brothers, bicycle 
' mechanics trying to get our contraptions airborne for long erious^ 
to demonstrate that their design is Sound', and we need not be 
• ..surprised that the onlookers are 'scebtical and show little 
enthusiasm for entrusting, themselve/ to the products 
(Nisbet, 1974, p. 18) y/ s ' 

• «t / 
The vital roles of evaluation in innovation include providing feedback of 
information to innovators about project strengths and weaknesses, providing 
a basis for discussion, and strengthening innovators ' capacity for planning. 
Nisbet also emphasises the' crucial role, of evaluation in the innovative ' 
process by claiming that: ... 



... any innovation which does not include systematic provision* 
for checking and reporting what has happened is pointless, a mere 
expression of protest, against the establishment. The poor; 
quality of evaluation is the weakest aspect of most of the present 
schemes for innovatibn, and,, using the metaphor of the hear»se, 
!• think it may prove a fatal weakness for many of these schemes. 

•(Nisbet, 1974, p. 26) 



PRESENT MONOGRAPH'S ORIENTATION 

A major assumption on which this monograph on evaluation is based is 
that evaluation and documentation are very important aspects of any* 
innovation.^* In fact, ' the two major reasons why innovators should engage 
in evaluation are clearly delineated in the Schools Commission's evaluation 
guidelines as being: 

to obtain information which will help the project director 
and the other participants to conduct ar*d develop the project 
more effectively and, later, to make decisions about its 
continuance. Information gathered during the course of the 
project should be put to practical use to improve the project. 

to -obtain information likely to be of value to teachers and 
Others working in similar areas or interested in trying out 
similar ideas. (Schools Commission, 1975b, p. 1) 

Furthermore, the essential gytfposes of tfre present monograph also "'are to 
provide assistance to innovators in tjjjj^at tempts , first, to collect 
evaluative information which will contribute to the improvement, smooth 
functioning and continuation of their innovations and, second, to report 
information useful in guiding others wishing to try a similar innovation. 

Another assumption of this report is that most people involved in 
school-level innovation will need som£ assistance if they -are to evaluate 
and document their projects Adequately. This assumption is supported by 
the observation that many of the final Reports written by Innovations 
Program grantees were found to be poor when files were scrutinized in 
Queensland (Maxwell, 197*8) L and in N;S,W. (Edwards, 1980). Similarly, the 
experience of Program counsellors who' visited grantees in N.S.W. to discuss 
evaluations attests to the fae* that the large majority of grantees lack 
knowledge, skills and experiehce in project evaluation. Also, Edwards 1 
(1980) survey of 35 Program counsellors in N.S*.W e revesyfed^that oyer 
90 per cent of these, counsellors thought that project directors were 
having at let.ast some problems with evaluation. Fui&hermore, although the 
Schools Commission's (1975b) evaluation guidelines had suggested systematic 
keeping of records, examination of 89 project^ directo.r^' reports on file 



revealed that only ahfcut 30 per cent had attempted any systematic record 
keeping, at all* (see Edwards , 1980) ♦ f ) 

Bepause augmented evaluation reports rjgpreseivt some of the few 



existing examples of comprehensive _evaluaxion and documentation of recent 
Australian innovations, they provide a ^6>tentiaJ. ly valuable source of 
guidance to other people attempting to render accounts of their projects. 
Augmented evafuatian reports, however, have had very low circulation, 
possibly because many reports are large and cumbersome.,'. Moreover, the • 
material in only a handful of augmented evaluation reports has been made 
readily, accessible in an economical way through publication as articles 
in national journals (see Power and M/Bryde., 1975; Fraser, 1977; Fraser 
and Koop, 1978; Johnstone and Fraser^ 1980; Parmenter and Fraser, 1980). 
Given that augmented evaluation reports largely remain a hitherto untapped 
source of valuable guidance to other innovators, it is highly commendable 
that th£ Innovations Program has commissioned the present report which 
conveniently draws together and makes readily available some of the material 
contained in augmented evsrfuation reports. 

This monograph is organised into two main ch&pters. In Chapter 2: 
Describing an Innovation.- examples ard drawn from augmented evaluation 
reports to illustrate some of the major aspects of an innovation (e.g., 
aims, materials, student activitie^, problems) which might be included 
when describing' air innovation. The types of information suggested in 
Chapter 2 usually can be assembled even if no specific evaluation techniques 
have been . employed , and are useful for disseminating ideas to others 
Contemplating a' similar innovation. 

In Chapter 3: Evaluation Techniques, concrete examples are given of 
the use of a wide variety of evaluation techniques (e.g., interview, 
observation*, questionnaire) which have been ,ysed in augmented evaluation 
reports. By using some of these techniques, innovators can enhance* their 
evaluations in two ways beyond that possible by restricting attention solely 
to the use of the descriptive categories outlined in Chapter 2. First, thfc 
use of these evaluation techniques appreciably expands the information base 
and enhances the description of an innovative project. Second, bjf* using 
these techniques to collect evaluativp information throughout the, life of a 
project, ad innovatoris provided with a. valuable fund of information upon 
which to base important decisions about project revision and organisation. 
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CHAPTER 2 : DESCRIBING AN INNOVATION 



, Several writers* on curriculum evaluation have claimed that an 
essential aspect of program,, evaluation is creating a description or 
portrayal of the 'program CStake, 1975, 1979; Kemmis v , 1977} Hall, 1979; 
Fraser, in press). In fact, Worthen (1977, p. 8) insists that ,f no 
evaluation Ls complete unles^it includes a%tfiorough, detailed descrip- 
tion of the program or phenomenon being evaluated. " In the present 
chapter, the focus is upon the identification- and illustration of some of 
the key features which might be considered wheji attempting to describe an 

innovation . — 

' V 

Whereas Chapter 3 considers specific evaluation techniques for obtain- 
ing additional information about a project, emphasis in this chapter is on 
simple features for describing. an innovation. Although not all of these 
features-' will be pertinent for a particular innovation, a description of - 
several of them can provide' a worthwhile fund of information upon which, to 
base the dissemination of a project. In particular, provision of a thorough 
description of several of these key features can- help others assess the 
applicability of an innovation to tfheir own situations and to provide the 
detail needed for effective replication of a project • # 

The approach followed in thi-s chapter involves drawing concrete 
examples and illustrations from reports of augmented evaluations of 
Innovations Program projects . It is hoped that this practical approach, 
because it is based on authentic examples- used in the documentation of 
recent school-level projects, will provide useful guidance to innovators 
wanting to give £n account of their, own projects. The seven key features 
which* are identified and discussed in this chapter are (a) aims and 
rationale, (b) setting, (c) origins of project, (d) materials and equipment, 
(e) student activities, (f) funding and (g) problems (e.g., staff mobility). 
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AIMS AND RATIONALE 



Aims and rationale are important aspects to consider when attempting 
to provide a description of any innovative project. Although the majority 
of augmented evaluation reports reviewed were found to contain a statement 
of the project aims or rationale, these statements varied considerably in - 
their degree of clarity, detail and specificity, 

Johnstone's (undated) Evaluation of a cyclical timetable projecf. 

(involving one day per fortnight of student activities or electives) 

provides the following simple and* straightforward statement; of aims: 

•# 

. -To expose school children to as much new knowledge and to as 
m&tjy new experiences as possible so that the pupils can 
% decide which of many experiences (or electives) they wpuld 

follow in the' employment of their leisure; 

To teach the pupil to be aware of himself, of others >and of 
the community as a whole. The main change to be brought 
about^is that education and learning are life-long processes. 
•(Johnstone, undated, p. 1) 



Similarly, Blackburn's (undated) augmented evaluation of several environ- 
mental education projects provides the -following informative listing of 
aims for one of the projects: 

. To establish a residential Environmental Study Centre at 
Binna-Burra on the edge of Lamington National Park; 

. To make the Centre available for use by primary, secondary and - 
tertiary study. groups ; 

\ 

. To provide suitable educational materials so as to maximize the 
educational potential of the national park; 

♦ ** 

. . . To encourage children to- be more aware of their environment, 
their place in it, and man s effect upon it; 

4 ."in the long term,, to raise the standard bf environmental 
education of the young generation; 

. For the Centre to be used as a model for future environmental 
study centres. (se*e Blackburn, undated, p. 2) 

In contrast to tljte simplicity of the above statements of aims, Searle 
and Parmenter's (1979) evaluation of a work preparation program at a special 
school .for mildly intellectually handicapped children reported a ntuch more 



comprehensive .two-part statement. The first part of the statement 
involved a description of the school's general rationale (referred to 
as the "value system 11 ), while the second part described the specific ♦ 
aims for individual schaol subjects compatible with the school's general 
rationale. The school's value system at the time of the project comprised 
the following belifefs held in common by the teaching stjj^f: 

. Handicapped people have a right to work. Their 1 ability to do so 
*- * -is not fully realized by^the community, nor is their right to do 
so fully accepted. 1 * | 

i j 

. The school <5an assist 'the handicapped to find wojrt^by involving 
itself more dLrectly in vocational education. 

. A work-experience program is an admirable vehicle for achieving 
this aim. Such a program has the advantag^of teaching employers 
about handicapped people. 

* . 

Our pupils haVe a right to education beyond the statutory leaving 
age, and for them to exercise this right is a decided advantage. 

Students need access to opportunities for continuing education; 
this could be catered for in existing institutions such as 
. technical colleges. (Searle and Parmenter, 1978, pp* 11-12) 

From these general rationale statements, the teachers at Minerva Street Special 

* r 

School produced a set of specific aims f or t each school subject. For 
example, the subject of Industrial Arts had the, fol lowing %lne specific t- * 
aims in the areas of occupational adequacy, personal adequacy and social^ 
competence : 

to develop skills in using the various tools, machines and 
devices available; 

> 

to develop good attitudes to and habits of work; 

^ » to develop habits of safety; 

to develop "basic skills in home maintenance and ^ar maintenance; 

. to develop an understanding of the properties and uses, of a 
variety of materials; 

to develop interests in leisure time activities; 

to develop an appreciation of design, an awareness of spatial 
relationships andean understanding of detail drawing in 
individual learning situations; ^ v 

* . to form an accurate assessment of individual abilities for 
vocational placement purppses;^ 

./* 

. to provide an environment in which individuals enjoy forking 
towards individual or group chosen objectives. (Searle and 
Parmenter, 1978, p. 82> 
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Numetous augmented evaluation reports describe details' of a project's 

geographic, architectural, personal and academic setting, and this provides 

/ a useful introduction to readers. In particular, because projects funded 

by the Innovations Program must be judged innovative in theit context, a 

complete description of a project requires consideration of setting.' 

A description of the projects setting enables the reader of a report to 

assess the relevance of that project to their own personal experience, 

knowledge, current concerns, students and situation. At the very least, 
* 

the inclusion of *a description of the setting can make a portrayal of a 

project jQQre interesting, concrete and colourful. MoreoVer, the inter- 
« 

- pretation of evaluation findings and their likely applicability to other 

contexts c#n fye enhanced considerably by describing the setting. 

» 

• Hogben's (1976) evaluation of a project based at a field study centre 
g>in*a historic reserve in Beltana includes the following most informative 
description of the project's setting: 

Beltana is a "ghost tovm" situated in an arid environment, 
some 530 kilometers by road north of Adelaide, in the Flinders 
ranges. The days are typically warm to hot while ^the nights are 
, clear and often very col4, It is a dry place; the annual rain- 
fall is around 200-250 mm (8-10 inches). Within easy reach of f 
Beltana are rugged hills, creeks (mostly dry), gorges, plains 
(variously covered), old mine workings, and t some grave-yards. 
Within an hour's easy drive to the north is the doal mining town 
of Leigh Creek; to the south is the small village of Parachilna. 

Ivan Hull and his family began their activities at Beltana in 
1970, haying searched South Australia for a site possessing 
buildings of historic value suitably for restoration, and an 
environment favourable for the establishment of a study centre for 
students. The purchase at low cost of several of the town's old 
buildings, and their custody over others 3s buildings of historic 
interest, provide the Hull family with accommodation (the old 
railway station and the police residence); a small income (a general 
store located in the railway station); buildings for the Study Centre 
(a small museum in the railway station, accommodation and sogse study 
facilities in an old hotel and the restored* school house); and the 
1 opportunity for the restoration and reconstruction of a place of 

considerable historic interest. (Hogben, 1976, 9. 3) 

Scheyer Templin (1978) has argued that still pnotography can make an 
important Contribution in educational program evaluation by illuminating 
a program's context. Still photography has been used to advantage in some 
augmented evaluation reports to elucidate project context. ,For example^ 
Boylan's (1977) evaluation of the .project at ^ournda Field Stddies Centre 
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included the photograph in Figure la to illustrate the appearance of 'the 
building and bush environment. Similarly Groundwater- Smith 1 s (1977) 

evaluation of a project at Carlton South Primary School provided the 

\S 

photograph in Figure lb to illustrate ihat the project was implemented 
in an old brick building set in spacious, pleasant grounds. 




Tig. la 




Fig lb 



FIGURE 1 # - Use of Still Photography to Clarify Context of Projects at* 

(a) Bournda Fi-eld Studies Centre (Boylan, 1977) and 

(b) . Carlton S?uth Primary School (jGroundwater-Smith, 1977) 
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Nicholas*s (1979) collection of evaluation, reports from the A.C.T. 
contains salient information about the directorlof each project being 
evaluated. For example, understanding -of the Canberra's Introductory 
English Centre" project for young migrants is enhanced by the following * 
fcolourful description of the project director: 

It was as a student volunteer that Edna Hopkins first became 
involved in teaching English as a Second- Language to refugees 
arriving in Australia from Germany aftd Austria just" before * 
World War II. After completing formal post-graduate teacher, 
training, she went to an Arabic-speaking country in the Middle East, 
and learned at first hand what it meant to have to fonction iji a 
new language. k A slightly different experience of b^ijig^an alien 
was afforded by the opportunity of living for some years in the 
West Indies - a member of a minority racial group... 

All these personal experiences contributed to a keen 
sensitivity to the needs of the. student who is learning English 
as a Second Language. In 1968, Edna Hopkins was the first person 
to be given a permanent full-time appointment in an A.C.T. school 
as an E.S.L. teacher... 

As a result of insights arising from experience, Edna has 
always stressed the importance of maintaining the mother tongue' 
and the cultural values which newcomecs bring with them, insisting 
that English, however necessary, is the second language. Another 
example of her forward thinking was the. dissemination of information 
concerning the school and the community and its social services in 
the languages of the student's whose English was still not sufficiently 
advanced to 1)e the means of accurate and effective communicatio-n. 
(Steen, 1979, pp. 29-30) 

Searle and Parmetffeer's (1978, ch. 2) evaluation report for the 
developing Independence Through Work Preparation" project provides an 
exceptionally detailed account of the setting of a project. In fact, in 
this report, the following classes of setting factors comprise the inputs 
in a systems model of evaluation: 

The school is described in terms of j±te adequacy of teaching 
materials and resources, the attractiveness of classrooms and key 
aspects of the school f s value system (e.g», the belief that 
handicapped people have a right to work). 

The student population is described in terms of the numbers, sexes 
and ^ges of students. Test data issued by the Vocational Guidance 
Bureau (e.g., I.Q., reading age, Purdue Pegboard Test scores, 
employability rating) are tabulated for individual students. Also 
student individual characteristics are summarised in a most 
informative descriptive paragraph for each student. For example, 
it is noted that Yvonne is a "controlled epilectic" and is "often 
the butt of teasing", that Raymond has a "most pronounced speech 
impediment and facial grimace" and is "willing and cooperative" 9 y 
and that William has a "happy and sunny disposition 11 and a "good « 
understanding of addition and subtraction". 
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The community in which the school is located was described in terms 
of the locality, other schools in the same shire, socioeconomic 
status and qualifications of community residents, and the results 
\ of surveys of employees 1 attitudes to employing handicapped persons. 

information is provided about existing community agencies (sheltered 
workshops, Commonwealth Employment Service, Health Commission, clubs) 
and facilities for recreation and continuing education. Also the 
parent body is descfibed in terms of attitudes to the role of the 
school* 

Economic factors affecting employment included information about 
general^ unemployment rates for people of different ages and 
sexes in various Sydney suburbs. (Adapted from Searle and 
Parmenter, 1978, ch. 2) 
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ORIGINS OF PROJECT 

Understanding of a particular innovation often -can be enhanced 
significantly by an appreciation o£ the project's historical setting. 
In fact, several augmented evaluation reports include a description of 
events xwhich occurred in the years preceding a request for Innovations 
Program funding, or of events which precipitated th^ conception of an 
innovative idea. The provision of these descriptions creates a 
historical perspective on a project which can help to explain important 
features and events associated with it. For example^ an overview of a 
large number of augment ^ ev aluation reports reveals that a sizable 
proportion of projects funded by the Innovations Program already existed 
in some form prior to the funding period. For these- projects , the 
Schools Commission gr^jit served to enrich or accelerate an* ongoing 
project* 

Searle and Parmenter f s (1978) evaluation of a work preparation 
program at a special school provides the following very detailed history 
of major events occurring over many years and leading up to Innovations 
Program funding in 1976: 

Following the initial survey in 1969, the school placed a 
greater emphasis on teaching social competence .since the lack of 
such competence was the most significant cayse,.of failure to cope 
in post-school life then. 

In 1971 the Conference of O.A* Principals f came to the conclusion 
that vocational preparation should be an aim of their schools. This 
school established a simulated work-situation in 1972 to cater for 
the need expressed above. Senior pupils were also tajcen on frequent 
excursions to real work places* 

In 1975 Mr Gary Roberts was granted leave for three months to 
investigate ways and means of better catering for the needs of the 
school's senior pupils. He found that employers held favourable 
attitudes towards the^ concept of work-experience programmes. 



22 



- 15 - 



He recommended that a programme modelled on those established in 
New Zealand would best suit. our needs. Further, he put forward 
the idea that pupils from the school would benefit from attendance^ 
at courses at a. Technical College. 

Parallel with the^e initiatives, Mrs Wirth's surveys continued 
to 'highlight shortcomings in the school's programmes, particularly 
with regard to the personal development of our pupils, the paucity 
of their recreational pursuits and opportunities, arid the instability 
they suffered in 'employment . 

In mid-1975 a work experience programme was established And has 
operated continuously since then. A report of the prdgramme was 
made late in 1976. 



In 1975, also, Mr Gary Roberts and Mr Harvey Ord prepared a 
submission for an Innovations Program grant to enable the school to 
develop further these earlier initiatives. This submission was 
amended by Roberts and Searle in 1976 and a grant was made to enabjle 
us to set up our current programme. (Searle and Parmenter, 1978, p. 11) 
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Lange f s (undated) augmented evaluation of an interdisciplinary open 
area program for Year 8 students provided the following informative 
description jof events preceding the receipt of an Innovations Program grant 

Towards»the end of the first semester 1974, the administration 
was approached by *a number of teachers with a request -to call a 
meeting of interested staff to .discuss the then Grade 8 program. 
This initial request had been prompted by informal discussion among 
staff who were concerned about students' introduction to the » 
secondary sphere and the attitudes that apparently developed about 
schooling during this beginning year... 

So the informal staffroom 'conferences' had successfully 
identified the problems, some vague notions 6f alternative 
procedures to overcome these, and a commi'tment to the necessity 
for more formal meetings so that the processes o^-mobi li2ing 
teachers and resources could be established to support the 
operationalization of these ideas. 

This commitment that was established oil, an organized basis at 
the administration-called meeting in mid-1976 was given further 
momentum when it became known that funds w£re available for 'grass 
roots' programs of this type from Schools Commission: Special 
Projects Program... 

Though an enormous amount, of time, effort, and planning had been 
committed during these early planning and development stages, the 
general feeling was that the project at its then level of conception 
could not # go ahead unless the funds were made avai tfeble. It was 
considered that the ^identified materials and resources were essential 
for the program to operate in a supportive and flexible environment. 
(Lange, undated, pp. 4-5) 

Connors' (1979) augmented evaluation of a women's electoral lobby 
careers brochure project illustrated the catalytic role of the Schools 
Commission Innovations Program for one group of innovators: • 
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One member of the WEL (ACT) Education Action Group was in close 
touch with the*"*Scho61s Commission, and was able to focus our 
attention on the possibility of applying for fundfe for a special 
project aimed at liflting the aspirations of girls who had been 
identified by the Commission as f an educationally disadvantaged 
group* . I feel now that this personal contact with the Schools 
Commission was significant. As a group, we 'tad not hitherto seen 
this link between us and >6e work of the School^ Commission. 

In November 1974/ the Educatfon Action Group of the Women's 
Electoral Lobby (ACT) received advi^l that it had been granted 
$3,000 to produce a booklet ... The brochure The Facts 'of Life 
was ah attempt to put before secondary school girls in Canberra 
# some -of the facts about women's working lives; to help girls think 
. realistically about the future; and to challenge some of the 
outdated myths*and stereotypes. (Connors, 1979, pp. 162-163) 

MATERIALS AND EQUIPMENT 

In many Innovations Program projects, a major use of funding is the 
purchase of materials or equipment. Consequently information about : 
materials used is important for those wishing to know about the project 
or to attempt something similar. Numerous augmented evaluation reports 
provide a rich source of information about project materials and furnish 
several good examples of how project materials might be described to others, 

Morison's (1978) evaluation of an innovative gross motor education 
program for moderately mentally handicapped children include an illuminat- 
ing description of materials. Figure 2 shows diagrams illustrating some 
of the equipment used by students, and the following verbal description 
is provided to accompany the diagrams: 

. ' 4 
l - Building Aid Climbing Pyramid is a tactile mountain of 14 pieces 

of foam rubber covered with vinyl and used to build a climbing 

hill for stimulation and creatiyjLty. 

V* 

2. Box Maze consists of ten five-p,ly boxes with sides out and ends out, 
or with a-shape cut in a side. The task is to negotiate the maze. 
Two children do it, with hands joined or elastic tied to their legs 
to make them realise that they must make changes in their movements. 

3. Tunnel Crawl Hoops are stitched into a tunnel so that children 
cannot see the end. The tunnel crawl can be moved to different 
directions and can be folded up. 

4 - Five Hinged Maze Walls Four or five sheets of three-ply, hinged 
and stood indifferent and varying positions, form a maze. 
(Adapted from Morison, 1978) ' . , 



24 



LBuilding and climbing pyramid" 

/ — 



v 



z 



4 



V 



no 



2. Box maze 



3. Tunnel crawl 




4. Hinged maze walls 




FIGURE 2 m Gross Motor Equipment for Moderately Mentally Handicapped . 
£PJ£ * • ' Children '(Morison, 1978) 
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Another example of a description of equipment i$ contained in' s% *~ / 
Blackburn's -(undated) evaluation of an environmental education project 
involving students in expeditions into' the bush.. For example Blackburn 
provides th& following informative list of equipment which was purchased 
with Innovations Program funds for use in each erf three expeditions: 

8x3 man nylon tents with" aluminium poles and pegs 
* 6 x 2- man nylon tents with aluminium poles and pegs 

18 x f H 1 frame packs - nylon --'Mt. King/ 1 
18 x^H' frame packs - nylon* 'Mt . Erica 1 
- 4'% : ,xomahawks wi£k carriers . 

4 x*machetes 

2 x stay sharp knives 

8/x 'silva' type 3 compasses 

♦8 x plastic water buckets , , 

8x1 gallon billies # ^WSi& 

1 x first aid kit - miscellaneous itjems >'?i^^*. * 
10 x plastic map -pockets . i'f> 

A listing of equipment such as this one can provide valuable guidance to 
.other teachers attempting to organise similar experiences for their students 

STUDENT ACTIVITIES 

A major feature of many projects funded by the Innovations Program is 
that they afford students the opportunity to engage in novel, innovative 
activities not normally experienced at school. An important part of a 
portrayal of thes-e projects, therefore, is to make clear to audiences the 
nature of these student activities. In this section, three good examples 
of thej portrayal of student activities are drawn^from augmented evaluation 
reports. 

"the Film Creativity Project of the A.C.T. Children's Film and 
Television Council 11 exemplif ies v an innovative project which involved 
students in an unusual activity, namely, filiq making. In iiis augmented 
evaluation report for this project, Fordham (1979) describes in detail 
several typical activities in which 'students engaged when making films. 
For example Figure 3, which is taken from Fordham 1 s report, shows a" film 
script on the' theme of "doomsday" created by ai group of Year 5 students. 
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FIGURE 3. A Film Script on the Theme of^)oomsday" created ,by a^ 
, Group of Year 5 Students (f roff Fordham, 1979, p. 64) 
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Another Innovations Program project in which students engaged in 
unusual activities was a mathematics project which involved students in 
^performing plays about the lives of. ancient mathematicians. Fraser and 
Koop f s (1977) evaluation reporfe contains a copy of one of these plays 
(entitle^ "Thales of Miletus") in* an appendix to permit ready usage of 
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the play t>y other teachers. Figure 4 shows an extract from this play, 
and clarifies how mathematics and history h&ve been combined into a play 
suitable for use by junior high school students. 
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They were built 
. two and a half 



THALES: Please, Anaximander . 

NUISERRA: Look behind you, Thales, look at our pyramids, 
over two and a half thousand years ago by our ancestors • 
thousand years ago. How high do you think they are? 

THALES: About the same length as one hundred and sixty paces. 

NUlSERRA: A good guess ... better still, how would you measure them exactly? 

. fife 

ANAXIMANDER: He's trying to discredit you. 

THALES: Are you asking me for a simple method of measuring the height*, or* 
do you also want me to measure' the height here v and now? 

MANETHO: * I suppose you want men and equipment to measure, the height, 

y . THALES : J I t's as imp 1 e me t ho d you wan t, isn't it? 

NUISERRA: Show us how you would do it, for no man has ever measured the 
height since they were built. 

THALES: I need only a man and some wooden pegs. 

ZOSER: (To Manetho), Do yot/ really think he can measure the height? 

MANETH6: Not a chance*. No one has ever answered the prolAem. The physical 
effort to measure the pyramid is just too great. The arrogant and conceited 
^always meet defeat with this question. Let's watch h'im squirm. 

THALES: Now, you shall ^see how* I would do it. t}m ... (Zoser and Manetho 
smile at each other and settle in their chairs, comfortably as if expecting 
a iong wait. Anaximander looks expectantly at Thales) 

THALES: Sorry ... sorry for the delay. 1*11 just' make myself comfortable 
on the ground, like this. (Stretches himself out) Anaximander ... 

ANAXIMANDER: (Puzzled but eager to do whatever Thales asks him) Yes, Thales, 
right here. (Whispers) .Are you all right? 

THALES: Yes, of course. Just> dp what I ask you, Mark out the length of my 
body. Put one^peg here at my head and the other at my feet. Have you done it? 
That's right, thank you. Now help me up. 

NUISERRA: Will you explain what you are doing, Thales? 

THALES: Certainly. My. student just marked out the length of my body on the 
ground. See between these two pegs. Now— if I stand here, right where my head 
was, and the sun casts a shadowof my body the same length on the ground, then 
the shadow of the pyramid is the* same length as. its height ... See.* 

ZOSER: What's that 'you're saying? 

THALES: I'll explain again. It's really very simple ... I lie down in the 
sand thus ... and my body makes an impression equal to my length. Now I 
stand up ... For goodness sake, Anaximander, help me* stand up . Now I stand 
up near where my head was, here. When my shadow, caused by the sun, is the 
same length as my body impression in the sand '(Points to sun and points to 
sand as he says the words), then the shadow of the pyramid is the same, length 
as its height - right? No - *don f t answer. ^ So all we have to do is £o wait 
for my shadow to be the same length as my body impression and, at the same 
time, measure the pyramid's shadow taken from the centre of its side. Right? 

ANAXIMANDER: (Big smile) Right! 

THALES: It's a right angled triangle - sides equal about the right* angle. 
The most important triangle there is. V 
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URE 4. Extract from a Play, "Thales of Miletus", about the 

Life of an Ancient Mathematician (from Fraser and Koop, 
1977, p. 53) 
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the "Environmental Study Centre 11 is a project which involved a group 
of students in the preparation of an environmental area. Blackburn's 
(undate.d) augmented evaluation, report for this project provides the 
following useful description of the activities engaged in as part of 
this project : • 

Miss Sonter f s workforce consists mainly of Grade 11 students, 
A nucleus of about twenty students spends around four hours each 
xgeekend working on the area. They are very enthusiastic about the 
I work and give of their time freely, even though they will not derive 
phe benefit of their labours - that will be a legacy'for future 
generations .of students . 

When the slash pines were planted a post hole digger was engaged 
because it was a rush job -to get them all planted. The ground is 
very hard -and digging holes manually is a very exhausting process. 
Since then however all holes have been dug by students ^and loam 
carted in to gjve the trees a start. All the trees planted h<ave % 
been donated by students and parents and there is no shortage' of 
further offers. 

Planning of the work to be done is decided upon by the students 
in consultation'with Miss Sonter. Besides digging holes and planting 
trees, they, have built several small log bridges across the creek, 
cleared the ponds of rubbish, cut steps into a bank and erected a 
t hand rail, set up a class area of log stffets with seating for forty, 
and built themselves a barbecue which they use about once a month 
after their working free... 

Most of the students who are engaged in working on the area 
have, 6f course, been raised in the city and, like most city children, 
are quite naive about many things, particularly in relation to plants! 
No attempt has been made to lessen the effort required in digging 
holes in hard ground - they have discovered that soaking with water 
helps make the' job easier .. . Many hours of labour were spent one 
weekend cutting a set of steps into a rocky bank. The end product 
was very functional but heavy rain showed the students the erosive 
effect of running water and resulted in much of their work being 
destroyed - a difficult way to learn, but very effective. 
(Blackburn, undated, pp. 7-8 & 11) 



FUNDING 

( 

There are two main reasons why provision of details of funding and 
expenditure is an important feature of a complete portrayal of a project. 
First, although project directors are responsible for arranging detailed 
auditing of their grants, discussions of funding contained in augmented 
evaluation reports provide general information to those concerned with'' 
accountability in the broad sense. Second, specific details about the 
costs involved, in implementing a specific innovative project can provide, 
invaluable assistance to otters attempting a similar innovation. 
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One of the problems inherent in describing details of a project's 
financial expenditure is the fact that Innovations Program funding quite 
often is used to support only part of a more comptehensive and integral 
project. A good example of this situation is described in the report of 
the e/aluation of a history of mathematics project at Vaucluse Boys High 
School : 

It is vi^al for a proper understanding of the present report, 
however, to appreciate that Schools Commission funds were deployed 
in supporting orffy some parts of a more comprehensive and ongoing 
project. Firstly, in terms of timing, Mr Grunselt had already 
completed substantial amounts of historical research and the writing 
of versions of some articles and plays prior to receiving the Schools 
Commission grant. Moreover, now that the Schools Commission's period 
of funding is terminated, Mr Grunseit has every intention of both 
modifying and" extending previous work. Secondly, while Schools 
Commission funds were spent specifically on the purchase of video 
and photographic equipment, considerable resources from elsewhere 
were required to support other aspects of the project. For instance, 
the Teachers Resources Centre provided support for the production of 
slide-tape kits, and the vast time commitment and modest expenditure 
on typing and stationery involved in the historical research and the 
production of articles and plays were supported either by Vaucluse 
Boys High School or the project director himself. (Fraser and Koop, 
1977, p. 4) 

A good example of an informative description of funding details iC 
shown in Table 1, which y is based on figures tabulated in Blackburn's 
(undated) evaluation report for an environmental project involving dis- 
covery, adventure arid survival in the semi-wilderness. This table^ provides 
an interesting comparison of the expenditure anticipated by the project 
director at the time of applying for the ^rant with the actual costs 
incurred. From this comparison, it can be seen that the actual costs 
exceeded those anticipated by an amount in excess of ^1,70*0, which is 
almost 40 per cent rfiore than expected. The biggest * discrepancy in cost 
occurred for bus hire, although there also was a reasonable discrepancy 
for food costs. 

Three features of this information about expenditure are noteworthy. 
First, as the Innovations Program grant was for $3,000, it was intended 
that students would pay a certain amount towards the costs of each expediti 
Consequently, this project provides another example of one in which Schools 
Commission funding was requested for partial support of a project. Second, 
this project illustrates that unanticipated circumstances (e.g., inflation) 
can give rise to unexpected cost increases that potentially can cause 

problems in project management. In the preseqt instance, the additional * 

3 

costs were covered by increasing the charge made to each student for the 



expeditions. Third, and most importantly, the information contained in 
Taoie 1 provides very useful information about likely costs for other 
teachers contemplating a similar innovation. 



TABLE 1. Itemised Breakdown of Anticipated and Actual Costs 
of Expeditions (from Blackburn, undated) 



Item 



Anticipated 
Cost 

$ 



Actual 
Cost 
$ 



Equipment 

Tents & packs 
Compasses 
Gas primuses 
Utensils 6c tools 
First aid kit 

Bus Hire , 
Food 

Publicity & Stationery 
Postage & Administration 
Films 6c Maps 



1,100 

30 
40 
30 



1,800 

1,200 

150 

150 

90 
4,590 



992 
26 
54 

160 
9 

2,979 

1,572 

58 

197 

282 
6,329 
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Lehman's (1978a) report of an augmented evaluation of a project 
involving an alternative to the traditional* faculty , classroom and graded 
study system also provides a useful description of expenditure. In 
particular, this description, illustrates how Schools Commission funding 
was used to support certain facets of an overall program, and proposes 
an Interesting system for categorising different coats. The Innovations 
Program grant . was for $12 f 715 altogether and this was used to fund an exist- 
ing program which was hampered by lack of sufficient and appropriate 
materials. A brief ifummary of Lehman's description of expenditure is given 
below: 
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Start-up costs The Grant funds were used to supplement the ongoing 
Year 7 and Year 8 operations on a ff start-up M cost basis. 

Operational costs were not supported by*Schools Confnission funding, 
but by us ing^fexi sting school funds and appropriate, generou? 
assistance from the school's Parents and Citizens Organisation. 

Management costs were borne by school personnel in their normal 
course of duty; 

Per learner s tar t-up^ costs The Innovations Program grant provided 
start-up costs- for the 600 students in Years 7/8 in 1976 of approxi- 
mately $9,300 for non-expendable items' ($15,50 per student) and 
approximately $3,400 for expendable items ($5.70 per student). 

Per learner operational costs The school, undertook upkeep of non- 
expendable items of equipment. However, in future years, it is 
expected that the module concept will require a budget of approxi- 
mately $6 per student in additio n to whatever the school has normally 
spent. For example, the effect of , the module upon expendable items 
in the Art, Home Science and Industrial Arts courses has been to add 
an annual increase in excess of $4 pe**student. 

Per learner management costs As the module was undertaken by staff 
in the normal course' of duty, there were no Innovations Program funds 
used for this purpose (e.g., teachers 1 salaries). (Lehman, 1978a, 
pp. 9-10) 
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It is desirable that reports of innovative projects include descriptions 
of problems encountered. Not only is an appreciation of problems necessary 
for a full understanding of a project but, more important ly 5 the identification 
of specific problems can alert other innovators (whether funded by the 
Innovations Program or not) about potential difficulties which could need 
to be overcome when attempting similar projects. While problems associated 
with specific innovative projects are described in particular augmented 
evaluation reports, this section identifies and discusser five general 
problems which characterised several different projects, These five problems 
are a) feelings of isolation, b) time delays^c) staff mobility, d) failure 
to establish need for an innovation and e) ineffective communication between 
innovators and others. 

Feelings of Isolation 

There is a certain amount of evidence contained in augmented evaluation 

t 

reports which suggests that innovators often experience feelings of isolation. 
For example,, the director of an Innovations 'Program project involving the 
development of history of mathematics material's explicitly expressed concern 



about working in isolation and receiving very Little feedback or con- 
structive criticism about his work (see Fraser and Koop, 1977, pp. 36 & 77). 
For this<|roject director, contact with the 'evaluation consultants through- 
out the a|gmented evaluation played a very valuable role in overcoming 
this sense of isolation and providing the support and constructive 
suggestions he was seeking. Beck and Goodridge's (1978) evaluation in 
Western Australia of 15 Innovations Program projects reTated to community 
involvement incorporated a two-day conference in which project directors 
came together to share their experiences. According to Beck and Goodridge, 
this two-day conference may have been the most important and beneficial 
part 6f the evaluation because it fulfilled a need for support and sharing 
of id«as. 

Connors' (19^9) augmented evaluation of the l&men's Electoral Lobby 
Careers Brochure project provides further tellingVvidence that a need 
for encouragement and support is strongly felt by people contemplating an 
innovation. In- particular, the fact that the Innovations Program officially 
funded this project provided considerable motivation to commence and 
continue the innovation. Connors notes that: 

The Schools Commission gave us a lot more than $3,000. It helped 
us see ourselves as innovators in our own community, and gave us a 
sense of our own power to influence change. In some ways, it is 
frightening .to admit that a group of people who have fared so very ' 
well comparatively in our education system need an official agency 
to tell them their ideas are worth something. How pathetically 
powerless must many others feel! * 

the money was important. It made the venture manageable for us 
and far less daunting. It also increased our own commitment. ' 
After aU^Thad been entrusted j?i th public money, and I Jcnow a 
number of us felt a responsibility to match it with our own time and 
energy... In this way, our project was a 'model' in terms of the 
philosophies and policies of the Schools Commission. The money was • 
not a handout - it was a catalyst. It unleashed energy and 
initiative gather than crushing it. Far from creating dependence 
on the Schools Commission", this grant was for 'some of us a useful 
temporary crutch we probab-ly no longer need. (Connors, 1979, p. my 

Time Delays 

According to evidence available in augmented evaluation reports,' - ' 
time delays can precipitate important practical problems for those engaged 
in initiating and maintaining an innovation . 'Although .Innovations Program 
funding nowadays is organised so that new grants , are 
available at the commencement of each new school year) this procedure was f 
not alway^ foLlowed for the earlier series covereTKo/augmented evaluation 
reports. Consequently , 'de lays between the time of submission of an 
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application and the grafting of funds sometimes gave rise to a variety 
of problems for specific projects. For example, when Innovations Program 
funds were made available for the coipmencement of an open area project in 
August, certain difficulties were experienced in the changeover to open 
organizational patterns'for students who had already spent two-thirds of 
the school year in more traditional surroundings (see Lange's undated 
augmented evaluation report). Similarly, Groundwater-Smith (1977) 
reported that delays between the commencement date 'anticipated by project 
personnel and the actual time of award of the grajit meant that some of 
the original project proposers had left the school', that the costs of 
some of the materials supported by the grant had increased appreciably, i 
and that some of the materials originally requested were no longer 
appropriate . 

An interesting example of a delay in progress on an innovative 
project is provided in Blackburn's ^(undated) evaluation report for an 
environmental education project involving the construction of a building 
which blended in and didn*t interfere with the rainforest environment. 
It was found that a delay of about six months was experienced in obtaining 
shire council approval for such a building. For example, a sign stating 
intention to build and calling for objections had to be erected on the 
site for 28 days, and certain councillors were keen that^his environ- 
mental centre should not be opened prior to another centre in the area 
sponsored by another organisation. 

Staff Mobility 

One of the most common sources of problems with Innovations Program 
projects is staff .mobility. In particular, quite serious problems can 
arise when key personnel involved in the initial conception and implemen- 

r 

tat^ion of an innovation leave the project (usually to take up" an appoint- 
ment in another location). Ingle's (1978) augmented evaluation of the 
Community Reading Centre project provides an informative description of 
the problem&„experienced when the original applicant withdrew soon after 
the funding-was made and when a further change of grantee occurred after 
the* project had been running forone year. The problem of staff mobility 
in this project is described by Ingle in .the following way: 

At the outset, the original applicant suggested that since she was 
unable to continue, it would have been better if the project had been 
allowed to lapse. In retrospect it may haye been wise to take this 
. advice. , Most projects of this nature depend greatly upon the 

involvement , .reputation and vision of the person originally involved. 
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In this'case, Mrs Harris was unique because of her work with the 
Cromer Reading Development Centre. When the project was transferred - 
to Mr Liddell there was bound to be some difficulty in that 
Mrs Harris was employed at the same schaol and had to stand by and 
watch the development of the project which she had originally regarded 
as^ hers . Further problems followed when it became clear that the 
Principal of Cromer School regarded it as "his". (Ingle, 1978, p.-4rf>) 

Similarly, Groundwater-Smith's (1978) evaluation of the Centres of 
Learning project illustrates that problems were experienced when the original 
project director and several project teachers left the project. In 
particular, the key problem was that: 

The staff implementing the project were not those who' had conceived 
% of it. There was a notable commitment on the part of two members 
of the new planning team, whilst others, had either teaching styles 
antithetical to the underpinning philosophy of the innovation or 
were less experienced and less secure as classroom practitioners 
(Groundwater-Smith, 1978, p. 5) 

Failure to Establish Need for an Innovation 

There is some tentative' evidence in augmented evaluation reports which 
suggests that the success of 'an innovation may depend in part upon the 
extent of t;he need established for the innovation in its context. This 
is certainly consistent with the intuitively plausible notion that a 
project is more likely to be successful where a clear need for the 
Innovation has been established than in a situation where the need for 
the innovation is dubious. 

Ingle's (1978) evaluation of a project involving a community reading 
^ centre provides a good illustration of a situation in which failure^ to 
establish a cleax need for the innovation contributed to its lack of 
success. Although a major component of the program was intended to cater- - 
for adult non-rea<jers , Ingle 1 s. report indicates that the area served by 
the project was middle class and that the number of adult non-readers in 
the area was likely to be 20 at the most. When this is. added to the fact 
that adult non.-readers are notoriously hard' to attract to classes (especially 
ones held in a primary school) , it appears that need for this project in 
•this area did not exist. This notion is well supported by the fact that 
only a single adult non- reader,, in fact, attended the centre throughout 
the project. Ingle claims that: • 

Although the submission was very thoroughly prepared, the one major 
weakness was in the survey of need for the existence of the project. 
Even a casual observation of the Cromer area would suggest that it 
\ ; ls not a "fertile ground" for adult n 0 n- readers. Furthermore, there " 

was provision for such people nearby in evening colleges and 
technical .colleges. (Ingle, 1978, p. 16) 
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•< In contrast, an apparently quite successful innovation involving 

resources in the areas of language, mathematics and craft was lopated 

in a school in whichthere were reasonably strong .grounds for ^assuming 
— I 

that many students could be stimulus deprived. In particular, evidence 
from Groundwater-Smith's (1977) report showed that 65 per cent pf 
students came from homes where the father's (or single parent's) 
occupation falls into the broad ^tradesman's category, and where almost 
half of the families had English as a second language at home. This 
evidence helps to establish that there was a need in this school for a 
project which involved the use of considerable resources for learning. ^ 

Ineffective Communication Between Innovators and Others 

Augmented^evaluat ions provide some gc^od illustrations of the 

importance of effective two-way communication between those involved 

directly with an innovation arid other people. There are^ considerable 
# 

differences from project to project, however, in the required levels of 
communication and in terms of the people (e.g., school principals, 
teachers, parents, community groups) with whom it- is important for 'project 
personnel to communicate. . v / 

There are several good reasons why two-way communication is important. 
First, by deliberately seeking information about. 'their projects from others, 
innovators are ^likely to have a mu^Jf-more informed basis for making important 
decisions about how to organise and modify their innovations. Second, there 
are interested groups of people (e.g., parents and other teachers) who 
deserve to be informed and consulted about an innovation. For example, 
Groundwater-Smith's (1977, p. 42) evaluation report reveals that a teacher 
who had v arrived in the school after the planning stage of the project 
perceived fl a definite policy of exclusion 11 and "being kept in the dark 11 . 
Third, because jealousy or resentment can arise sometimes -between people 
excluded from a project, acceptance and understanding of the project by the 
school principal, other teachers or parents can be critical for the 
successful implementation and eventual continuatipn of a project. 

An example of an innovation in which the project director took special 
-steps to achieve -effective communication with parents is an environmental 
education program -evaluated by Blackburn (undated). Because some parents 
were somewhat hesitant about the idea of their children going on 
expeditions, the project director required the parents of each student 
going on an expedition to attend two briefing sessions. These sessions 
clarified aims, rationale and organizational details, involved the showing 
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of slides and movies of the type of country involved, and enabled the 
parents to get to know the project director. Blackburn's augmented 
evaluation report shows that these briefing sessions were very success- 
ful in dispelling parents' fears about the expeditions* 

Brady's (1979) augmented evaluation of an aboriginal community 
education project is an example of a project for which less than optimal 
communication with other teachers and with some parents detracted from 
the* overall success of the project. Although this project was clearly 
successful in the eyes of the aboriginal children and their parents, many 
teachers and the parents of white children felt that the project could 
seriously disadvantage white children. According to Brady (1979, p# 19)^ 
a possible atea in which this project director might have failed was 
"in not recognising the need for much fuller consultation with her own 
colleagues and with some of the parents of the white childrW." Furthermore, 
this lack of understanding and acceptance between the project director 
and her teaching colleagues mitigated against the establishment of the 
innovation as an on-going part of the school program for the following 
reasons : 

The reasons given for this were that the informal nature of the 
activities clashed directly with the assumptions of the teachers 
as to how classrooms should operate. The activities were seen as 
not being serious as compared with school work, and children were 
sometimes not 'given permission to attend the activities as a punish- - 
ment or to enable them "to catch up on school work". It was found 
that improved participation by the Aboriginal children in decision- 
making in relation to their own lives was the last thing that anybody 
in the school wanted, the pattern there being of having orders handed 
down from the top - principal to teachers to children. A program 
which encouraged individual thinking about decisions was obviously 
counter to the school ethos, (Brady, ^1979, p, 12) 

Beck and Goodridge's (1979) augmented evaluation report describes an 
innovative project in which project personnel recognised fully the 
importance of communication between the school £nd parents: 

The problem of communication between parents and the school is 
evident in most areas but it had become of crucial importance in 
this small non-Government school, in a semi-industrial area o# the 
outskirts of the capital city. The majority of pupils come from 
homes where only Italian or Yugoslav or a mixed dialect is spoken 
and where parents tend to be conservative arfd culture-bound in their 
attitudes towards school. None of the teachers could speak the 
languages Spoken by these migrant communities, (Beck and Goodridge 
1979, p. 11) 6 ' 

This communication problem was exacerbated by the fact that notes sent 
home to parents in their own language often failed because many migrant 
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par en ts could, no t _r ;e _ i n_ any langu age . Mo r e oy e r ^ _ t.h e _p_r o_ j_e,c_t _d ire c tor 
,was aware of a certain amount of hostility among parents who feared 
that many of the ideas and methods being used at the school were 
unrelated to the learning of essential basic skills. The way that 
this communication gap' was successfully bridged was to use Innovations 
Program funding^to employ a full-time liaison person who was acceptable 
to and respected by this community, who visited migrant families 1 homes 
and who was available at the school to talk to parents whenever they 
wished to come. 



" — CONCLUSION 

This chapter attempts to assist innovators in giving an account 
of their own school-level projects by providing examples of how to 
describe the following seven key characteristics of a project: 
aims and rationale; setting; origins of project; materials and 
equipment; student activities; funding; and problems. By giving a 
description of an innovation based on several of these features 
(which normal J. y would not entail the use of the specific evaluation 
techniques discussed in Chapter 3), an innovator can provide 
worthwhile assistance to others wishing' to implement a similar 
project. 

By identifying and describing problem areas associated with a 
particular innovation, a project director can help other innovators 
to /become aware of and ultimately combat these problems. In the 
present chapter, five such problems threatening the effective 
implementation and continuation of innovations were identified by 
reading augmented evaluation reports. These problems were a §'ense 
of isolation, time delays, staff mobility, failure to establish need 
ftfr an innovation, and poor communication between innovators and 
other people. It is noteworthy also that Huckins and Bernard (1977) 
have recognised the importance of effective communication and human 
relations when conducting an innovation, and that Porter's (1980) 
survey of 207 people involved in 95 Innovations Program projects 
in Western Australia confirms that staff mobility was a major factor 
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imbuing the implementation of. their.. innovations.- Moreover, there 
is. considerable scope for extending this chapter's initial attempt 
at identifying common problems by conducting a comprehensive survey 
of how^ widespread these and other problems are among a larger group 
of people engaged in innovative projects. 
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CHAPTER 3 : EVALUATION TECHNIQUES 



Whereas the previous .chapter outlines key features for describing an 
innovation, this Chapter discusses evaluation techniques which can be used 
to enhance -project documentation. Moreover, by Employing some of these 
techniques during a project, an innovator canjrfDtain a valuable information 
base which can guide the ongoing operation and improvement of an innovation. 

This chapter draws on the reports of evaluations of other innovations to 
provide examples of the use of eight potentially useful evaluation techniques, 
'^hese techniques are (a) interviews, (b) observation, (c) student diaries, 
(d) usage rates, (e) expert testimony, (f) discussions at meetings of 
project personnel, (g) achievement tests and (h) questionnaires (including - 
published instruments, home-made" instruments, Likert scales, semantic 
differential, and checklists). Furthermore, because various evaluation 
techniques will not be equally useful in the evaluation of any innovation) it 
is important that innovators "judiciously select those evaluation techniques 
most suited to tfoe nature and purposes of their particular projects. 



INTERVIEWS 

One of the most common sources of evaluative information used in 
augmented evaluations was interviews conducted tfith project personnel, 
students, teachers and others. For example, Groundwater-Smith (1977) 
tape recorded the answers of the resource teacher, the E.S.L^ teacher 
and the teacher librarian involved in the Centres of Learning project to 
specific questions such as "Are there any particular satisfactions or 
complaints which you might like to voice?" Sotne of the illuminating 
comments made by the resource teacher in response to this question are 
given below: ' 

The centres are located at one end of th£ building, so that 
teachers of senior grades more remote from them and who feel 
some need of surveillance are reluctant to send their children 
any distance to work by themselves. Also lack of carpeting and 
. acoustic tiles made the area relatively noisy ~ a faotor some 
teachers found intolerable. Finally, because implementation was* 
not in the hands of the original planners, the whole project 
could be seen, in one sense, as being an "imposition 11 on staff 
(Groundwater-Smith, 1977, p. 42) 
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LangVs (undated) augmentefrevaluation of aa^Sfharea 



involved taped interviews with project teache 
furnished informative comments including: 



rs , 




program 



ese interviews 



. Smaller spaces or withdrawal rooms are necessary f 0 r tutorial 
.discussion sessions, taping, television viewing, etc. 

• Open Area has a higher noise level and more movement than 
is evident in more traditional .settings . * 

t . Teaching in an open area requires a blend of effective classroom 
management, and the ability to treat students warmly and humanely 
in a fairly informal situation. fc - 

. Ail teaching in the- open area is 'busy time'' - there is a need to 
respond to student .requests , but also to seek out the quiet 
student who may be needing help, (see Xange, undated, p. 19) 

Hogben '.5,(1976) "evaluation, of. a project at a field stuies centre 
also involved interviews, which were conducted by telephone, 'for convenience^ 
and to. save time. . Examples 0 f some of the interesting, comments ,made by. 
teachers included: X ' 



. The way it s run be&ts anything I^ve ever seen: it's "really 
well run. < < ( 

. That which -is achieved at the canfp in terms of knowledge* of arid 
, and semi-arid land environment, living within an ecosystem and 
adapting to that ecosystem, and living as a co-operative unit of 
humans within harsh, conditions deserves the .highest recommen- 
dation and praise. ' . 

* l ^ 

. The atmosphere for social interaction was unsatisfact6ry and the 
experience futile from our point of view - either as h social 
experience, or a geographical field exercise. (Hogben, 1976, p. 7) 

An example of # ,an augmented evaluation which made use of student 
interviews to provide- evaluative information is Cassin's (1977) report of 
a multi-media geography project. For example, these intervi§w? v with 
students revealed that: group discussion and glroup activities were enjoyed 

by most students; five or six groups of five in the room at the same time 

ft • * 

does'not interfere with gro'up activity; and-students feel they can learn. 

. * V * 
more through the inquiry method. - 
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OBSERVATION 



An overview of augmented evaluation reports reveals that, for the 
large majority of evaluations, the ^valuation consultant visited the 
project site in order to observe the project* For example, Robertson's 
(1978) evaluation of a multi-media centre project involved a total of 
nine site visits, and Lange's evaluation of open area programs involved 
regular half-day site visits in which the evaluation consultant acted as 
a participant observer. This process of observation of project operation 
often provided new evaluative information not otherwise available or 
furnished information which complemented evidence gained by other 
procedures (e.g., questionnaire survey). While most augmented evaluations 
employed an unsystematic and unstructured approach to project observation, 
a number of reports illustrate the potential usefulness of following a 
more systematic procedure in certain situations. 

Teasdale f s (1978) evaluation of an~innovatxon-_fO£ hearing impaired 
children involved an important component of observation. , In particular,^ 
because the speech of these children was retarded^ to varying degrees, 
observation of facial expressions formed an essential part of ^Teasdale 1 s . 
evaluation of the students 1 communication skilttfT~*~ J Fefh*dale described 
these facial expressions in phe /ql lowing way:/ 
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The whole gamut of human emotion/ could be^^ggetii't various times in 
the faces of the hearing impaireq^j^ildrenr^^ruzzlement^, gradual 
amusement, spontaneous pleasure,^ anger, offence, misunderstanding, 
joy, hurt, confusion, excitement, disdain, jdislik'e and so on were 
readily observedj and acted as a most effective dommunication means. 
One child whose speech was particularly retarded showed an 
exceptional ability feo communicate not only through facial 
expression,, but through total body expression, -(feasdale, 1978, 
pp. 21-22) \ 

Groundwater-Smith's (1977) augmented evaluation of the Centres of 
teaming project provides an example of the application of systematic 
procedures for observing and record ijig^-&etferal aspects of classroom 
organisations communication and behaviour. This type of observation data 
provided quWtitatiye evidence about the extent of implementation of 
intended innovative strategies in classroom individualisation/ These * 
systematic observations wete made by university students working 
in small groyps to measure and record classroom events. Prior to the 
observation period, .these observers were trained using videotapes, and 
adequate inter- observer reliability was established, 
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The group of university students observed four different classes on 
each of three separate occasions. A "dimensional occurrence scal^" was 
used ip observing and recording several salient features of classroom 
behaviour or organisation over intervals of four minutes 1 duration. For 
example, the number of minutes within each four-minute interval devoted to 
each of four classroom grouping practices (whole class, small groups, 
individual students, individual students pips small groups) was observed and 
recorded as shown in Table 2. An example of the interpretation *of the data 
in this table is that in the third class (Cl^ss 5/2) during the first four- 
minute observation period approximately three minutes were devoted to whqle 
class activities, while one minute was devoted to small group activities. 



TABLE 2. Number of Minutes Spent in Different Grouping Arrangements 
on Three Four-Minute Occasions in Four Classes (from 
. Groundwatfer-Smith, 1977) 



Class 


Occasion 

f 

• • 


Number of 


Minutes 


Spent in 


Each Grouping Arrangement 


Whole 
class 


Small 
groups 


V 

Individual 
students. 


, Small groups 
& indiv. students 




Fi/st 


0 


0 


0 




4 


6/6 


Second 


0 , 


-0 


0 




4 




Third 


4 


0 


0 




o' 




* 

First 


2 


2 


0 




0 


5/3 


Second 


3 


» 0 


0 




1 




Third 




0 . 


0 




1 

0 




First 


3 


1 


0 




0 


5/2 


Second 


4 


0 


0 




0 


i 


Third 


2 


0 


0 




2 




First 


0 


0 

v. 


0 






Craft 


Second 


0 


0 


.0 




4 




Third 


2 


0 .. 


0 




2 
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^ Videotaping was employed as a way of recording observations for later 
processing in Morison's (1978) augmented evaluation of a motor stimulation 
project for moderately intellectually handicapped children attending a 
special school. The project involved students in the use of gross motor 
equipment (e*g., a building aid climbing pyramid, a box maze, a tunnel 
crawl), and one of the main Outcomes studied in the evaluation was improve- 

ment over time in student proficiency in performance of selected activities. 

# 

The evaluation consultant visited the school on five occasions to 
videotape a set s of six activities which were 5 perf ornted by each of nine 
children. The visits were ma,de at- two-ajtoth intervals and the six 
activities were net climbing, foam bloolo-Ta Iking, beam. walking, ladder 
climbing, trampo lining and carpet rolling. All performances at each 
session were video recorded in their entirety using a single, tripod- 
mounted monochrome camera and natural classroom lighting. 

For each of the six activities, each student's videotaped performances 
.on the fjbcS different occasions were arranged into random order, A group 
of expert judges then viewed each student *$ random sequence of five 
performances in order to provide their opinions about the ranking of 
efficiency of performance* These rankings were averaged across judges 
and across the six activities. This enabled the actual temporal order of 
performing the set of activities to be compared with the order of average 
rating of efficiency of performance (as perceived by the experts). Data 
for each student were subjected to a statistical significance test in 
order to ascertain whether the student's efficiency of performance (as 
perceived by Expert judges) had improved over the time that video records 
had been collected. Evidence indicated that,of the nine students, six 
showed some evidence of improvement in performance over the time of the 
evaluation study. fc . r 

STUDENT DIARIES 

■» 

A simple way of maintaining records of evaluative information is to 
have students involved* in an innovation keep diaries. Blackburn's (undated) 
augmented evaluation of a project involving expeditions into the semi- 
wilderness required students to ke^p diaries which yielded a large amount 
of interesting and informative details about student perceptions of this 

project. Examples of some extracts from student diaries are provided below. 

4 
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This morning we woke pp to frozen packs. Layers of ice covered 
the ground and our boots. Groups 3 and 4 f s plates and billies 
were full of ice and their cutlery was stuck in it. After getting 
everyone up we took phonos of Robyn's frozen pants and washer. 

The most amazing feature of the gorge was its unspoilt grandeur. 
Man has not touched it in any way*, We, as explorers, had a feeling 
of being pioneers and remembered Ludwig Leichhardt in his loneliness. 

The view could only be described as one of the truly magnificent 
views that I have ever witnessed and the sight will stay with me 
all my life. We claimed' the record for the largest party to have 
climbed the peak. 

Arcadia enabled many young people to get together and started 
friendships which will last for many years. The loss of Arcadia 
is something which should be prevented. As, I sit alone on the last 
night I let my mind roll over the events of the previous nine days 
and I feel fantastic. Without any doubt these have been the best 
days of my life. I will remember them for ever, (from Blackburn, 
undated, pp. 23-29) 



USAGE RATES 




Usage rates which describe the frequency with which an inna 
iVlTF^iised can provide telling information about certaiti kinds of projects 
In particular, the success of projects without a captive audience can 
judged in part in terms* of the numbers and types of clients attracted 

* V 

This follows from the obvious fact that the beneficial effects purpor 
to result from a project can only be realised if some clients do experience 
the innovation. Although usage of an innovation is no guarantee that * 
hoped-for benefits will accrue,^ usage is nonetheless a necessary condition 
for the success of an innovation. 

The Innovations Program project at the Bournda Field Studies -Centre 
provides a- good^ example of a project whose success could be evaluated in 
part in terms of usage , rates. Table 3 shows some of the usage 

rates included in Boy lan s (1977) augmented evaluation 

report for this project. This 'table indicates that an appreciable number 
of school 'children (2499 in total) visited the Centre during 1976 and 
Term 1, 1977. Another pattern evident from the table is that/ compared 
with 1976, there was a , marked increase in the proportion of primary 
schools visiting the Centre. 



TABLE 3. Usagfe " Rates f or the Boumda Field Studies Centre 
(adapted from Boylan, 1977) 



/ A % 9 

Usage Rate 

Aspect _Z m 

klU of 1976 Term 1, 1977 



Number of visiting primary schools 


30 


20 


Number of visiting secondary schools 


27 


6 


Number of visiting students 


1684 ,c 


815 


Mean number of students per school 


30 


31 


Mean duration of visit (in days) 


- '1. 4 


1.5 



Boylan alsb reported valuable statistics about the localities from 
which students visiting the Field Studies Centre Were drawn. Of the 57 
schools visiting the Centre in 1976, 32 were from the local townships of 
Bega and Tathra and only seven were more than 80 kilometres away from the 
Centre. Similarly, of the 26 visiting scho^s yi^Term 1,^1977, 16 were 
from either Bega or Tathra and 4 only fivewere from outside an 80 kilometre 
radius. This information highlighted the fact that the Centre's facilities 
were being used by only a relatively small fraction of South Coast schools, 
and suggested the need for more effective publicity about the Centre's 
facilities. 

Furthermore, observations made by the evaluation consultant and 
discussions with the teacher in charge of the Centra revealed that the 
, Centre's facilities were being utilised by a broader cross-section of the 
community than originally anticipated. Boylan ^notes ^that: 

Community groups, such as the Boy Scout Association, are making use 
of the tracking system on weekends, and individual members of the 
community were using the tracking systfm during their vacation times. 
(A case in point is that one member of ° the P*eakhurst fttgh School group 
had previously camped within the area with her parents on several 
occasions prior to her visit with the school). Additionally, while 
the evaluator was at the field Studies Centre,^ film making group 
- from Melbourne arrived to make a full length feature film using the 
Bournda area as the site for their outdoor film sequences # Other 
weekend use of the facilities developed through the Conservation 
Society includes a weekend_visit by science trainee teachers from a 
Sydney-based College of Advanced Education, (Boylan, 1977, p # 34) 
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In contrast to the Field Studies Project evaluation, Jackson's (1976) 
evaluation of a toy lending library excluded collection of any usage 
rates. Yet. such statistics could have provided the library's 

t 

organisers with valuable information about the popularity of various items, 
the age range 6f borrowers t frequencies and durations of loans, etc. 
Such information could have helped to identify future purchasing and * 
advertising needs. 

Another example of the way that usage rates provided telling 
information about a project is contained in Ingle's (1978) augmented 
evaluation of the Community Reading Centre project for adult and ^ 
adolescent non- readers. Although the project was successful in attract- 
ing adolescent non- readers, only one adult client was attracted despite 
reasonable publicity. Consequently, the intended aims of the project 
had no chance of being realised for adult non-readers as this clientele 
was not reached. This evidence also suggests that adequate need for this 
project may not have existed when the project was proposed and funded. 



EXPERT, TESTIMONY 

A very valuable source of information about projects can be obtained 
by seeking the testimony or opinion of selected people with expertise in 
areas relevant to the innovation. Despite its potential usefulness, 
however, the great majority of augmented evaluations have neglected this 
informational source. An example af one evaluation which made use of 
expert testimony is Fraser and Koop's (1977) study of a project involving 
development of instructional materials related to the history of 
mathematics. In fact, some of the instructional historical materials 
were sent to various experts who were requested to provide their opinions 
in writing. 

A group of five experts - a historian .of mathematics, two mathematics 
educators and two drama critics - provided their views on a play "about 
Thales, an article aboutTconics , or both. Each of the experts was free to 
comment on any aspect of the material they wished ratjier than being requested 
to respond to specific questions/ Nevertheless, each expert was 
specifically requested to provide -opinions about the overall efficacy and 
* usefulness of the materials (summative evaluation) and to Identify any 

specific weaknesses which might be improved in any future rewritten versions 
of the materials (formative evaluation); 
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The comments made by each expert were "listed in full in an appendix 
in Fraser and Koop's report. For example, the historian of mathematics 
made the following illuminating comments: 

. The. play is considered valuable and should be published 
for school use after modification. 

. The ^assertion that Thales was exactly 60 years old in 540 B.C. 
is more precise than present historical knowledge permits. 

. Greeks of this period did not measure angles in "degrees" as 
suggested in the play. 

In addition to modifications suggested by the above comments, other 
positive suggestions were to include a few historical notes (disclaimers) 
with the play so*that the reader wouldn't take everything too literally, 
and to add suggestions about possible discission and field work which the 
teacher could conduct after tifie play had been used. 

Some of the salient copnents made by the mathematics educators are 
summarised below: 

% The idea of bringing the history of mathematics into 
the classroom is laudatory. 

* n 

9 The Thales play may be more suitable for use as a follow-up to 
work on similar triangles than as an introduction to the topic. 

In-service education would be needed before teacher^s^ 
could put the materials to optimum use. *' ^ 

Examples of comments made about the Thales play by the drama critics included: 

•> 

. The play has a number of positive dramatic qualities including 

its dialogue, settings and movement. 

• -* 

The play would be relatively easy to produce. 

. There is a lack of parts for girls. 

» * * 

p The flavour of the testimony solicited from the historian of 

mathematics, and the way that this testimony provided much useful guidance 
for improving the mathematical play, is illustrated in the following 
excerpt taken from an appendix %.o Fraser and Koop's report: 

,t# * 

Perhaps the first thing to be mentioned is the dubious nature of the 
source material... The latest short account of Thales is in the fine 
Dictionary of Scientific Biography , of which about 13 volumes have so 
%,far appeared. Here' the 1 dates of Thales are briefly discussed and the 
best estimates are given as 625 (?) and 547 (?) . Soothe assertion 
that Thales was 60 in 540 B.C. is too precise (page 1 of play). 

Most importantly, some reconsideration of the . two 'applications of 
- 1 similar triangles is in order. If Thales' shadow were a half or a 
q third as long as Thales is high, the pyramid would cast no shadow 

owing to its 'shape,.. „ As for the distance of the ships being equal 
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to a measurable distance along the beach (need to keep both base lines 
horizontal), a five foot staff of uncertain perpendicularity would be 
a quite unsuitable basis for any more reliable data than any 
experienced look-out might reasonably be expected to supply by looking 
out to sea. At least Thaies could stress how important it is to keep 
the staff vertical. 

No Greek ever used "degrees" for measuring angles, so far as any 
record survives , until well into the Hellenistic Period... And, of 
course, the suggestion that any scale. of angles would need to enter as 
intermediaries is quite unnecessary (and historically misleading). 
The natural thing would be mark the scale directly in distances. 
Needless to say, no precise information survives, as to what methods 
were used, (see Fraser and Koop, 1977, pp. 70-71) 

The testimonies sought from experts by Fraser and Koop were found to 
support results obtained in the evaluation from other <Iata sources 
(e..g., questionnaires). But these testimonies also raised several new and 
important ideas for the project director to consider when redeveloping the 
materials . 

\ : * ^ 

K 

N * DISCUSSIONS AT MEETINGS OF PROJECT PERSONNEL 

A very fruitful but neglected method of obtaining evaluative infor- 
mation about an innovation is to docume^ discussions during meetings 
involving project personnel. Such meetings could involve either the various 
people involved in a particular project or a group of project directors 
involved in distinct projects on a related theme* The essential feature of 
this relatively unstructured approach is that these meetings act as 
"brainstorming" sessions in which a wide variety of information can be 
obtained about a project's major strengths, and - shortcomings and its 
unanticipated side effects, and suggestions can be made for its^mproyement . 
In particular, matters raised spontaneously by participant^nay highlight 
problems they are facing in t&e implementation of their innovations , provide 
anecdotal evaluative evidence and stimulate the search fc/«a possible solutions, 
Mbn&ayer, this information then becomes available to others .contemplating 
work ,in ds^imilar arda and may warn of possible trouble spots to, be avoided 
in future planning. \ ^ 

B^ck and Goodridge (1978) used discussions at a meeting of project 
directors in their thematic approach to the evaluation of 15 school-based 
projects exploring community involvement in schools in Western Australia. 
As part of their evaluation, these workers organised a two-day conference 
involving the directors of these projects. Experience gained from this 
conference led Beck and Goodridge (1978, p. iii) to claim that "for project 
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directors this may well have been the most- important part of the study:, 
as it permitted them to evaluate their own projects in the light of the 
shared experiences and observations of all the participants," 

Based on comments made during this two-day conference, Beck and 
Goodridge were able to assemble a list of positive features that had been 
found associated with many of the projects dealing with school-community 
involvement. These included: ✓ 

It is beneficial for children to have contact with a variety of 
adults in addition to their teachers, 

. Students need to be and feel les£ isolated from the community 
they are going to work and live in. f 

. When there is a two-way understanding between home and school, 
both can work in the one direction for the educational benefit of 
the child, (see Beck and Goodridge, 1978,- p, 20) 

Discussions of project directors 1 experiences also lead to the 
identification of important problems and constraints which were associated 
with projects attempting closer ties between school and home or school and 
community. The identification of these problems pould provide cons iderable 
guidance to others implementing community involvement projects. These 
problems included: 

. Project directors who planned to involve parents during school 
hours in helping at the school found that the 'thousand's of 
bored housewives 1 were not there. Relatively few parents were 
in fact^available for any sort of involvement during school hours. 

. The presence of additional adults around the school increases the 
scope for personality clashes. For example,^ there are no easy 
procedures for dismissing unwanted and unsuitable* volunteers . 

* School buildings tend to lack facilities for parents, who thus 
feel awkward and redundant unless they ar§ actually working in 
a classroom' or library. 

. The time and effort spent by a teacher in organising and helping 
visiting experts can outweigh the benefits gained* (see Beck and 
Goodridge, 1978, pp. 21-22) 



ACHIEVEMENT TESTS 

For those innovative projects for which students' cognitive achievement 
in certain areas is an important aim, it is desirable to use either 
"home-made" o v r published tests to assess the extent of achievement. Also 
it is common in this situation to administer the same cognitive test as a 
pretest and as a posttest in order to gauge the amount of change in 
achievement occuring during the time of a project. erf) 



Lange's (undated) evaluation of a remedial mathematics project used a 
published test developed by Elkins , Andrews and Cochrane (1972) to assess 
changes in mathematics achievement among students involved in the project. 
This published test measures basic skills in arithmetic, particularly basic 
operations with whole numbers, fractions, and decimals and the application 
of these operations to measurement concepts. This test was administered as 



^a-^pretest in February prior to the commencement of the remedial program 
and again as a posttest towards the end of the year. Also, in w order to 
permit more meaningful interpretation of findings, these results were 
compared with those of a control group of students not involved in the 
remedial program. When Lange performed a number of statistical significance 
tests on these data sets, it was evident that the remediation program had 
been successful in promoting cogni tive* mathematics achievement among students. 

An example of the use of a "home-made" achievement test is provided by 
Fraser and Koop's (1977) augmented evaluation of a project involving 
development of some history of mathematics materials. In particular, since 
an. important aim of a play about an ancient mathematician was to convey 
the conceptTof similar triangles, a test was constructed by the external 
evaluation consultants to measure understanding of this concept. In order 
to gauge the effectiveness of the play in promoting understanding of 
similar triangles^ this test was administered to several classes of students 
prior to using the play and again as a posttest after completion of the play. 

The results of State examinations sometimes can be employed as indices 
of cognitive achievement in the evaluation of certain projects. For example, 
in his evaluation of a cyclical timetable project, Johnstone (1977) reported 
an imaginative use of Higher School Certificate results. Although a rise 
in academic standards was not an aim of this project, an expectation implicit, 
in the introduction of the new timetable was that academic standards would 
be maintained. Consequently, Johnstone gauged the* effect on academic 
standards of the cyclical timetable project by comparing the results bf 
students presenting for the Higher School Certificate before 'the organisation 
had been changed with results two years after the new oxganisation ^as 
implemented. It was found that standards of academic performance at the 
Higher School Certificate level had been maintained, arid that the variation 
between the results for the two years was very flight. 

Similarly Cordwell's (1977) description and evaluation of the Multi-Media 
Geography Laboratory project involving inquiry approaches made use of results 
awarded by the Board of Secondary School Studies. In fact, Cordwell provided 





a table showing the percentage frequency of seven levels of results for 
Year 10 Geography at the project school in the year prior to the innovation, 
in the year of introduction of the innovation, and during two successive years 
when the -innovation was in full swing* These results showed that there had 
been a greater frequency of higW&r achievement levels during the later two 
years, thus suggesting that the innovation in geography education was having 
an advantageous effect on student achievement. 

QUESTIONNAIRES 

One of the most frequently employed methods of collecting information 
in augmented evaluations was the use of questionnaires. Moreover, augmented 
evaluation reports provide numerous worthwhile examples of how a variety of 
different types of questionnaires have been used to advantage in the 
^ \ evaluation of innovation, tn this section, questionnaires are discussed 
using the headings of published instruments, "home-made" questionnaires, 
Likert scales, semantic' differential , and checklists. 



ERIC 



Published Instruments 

Relatively few augmented evaluation reports made use of ^published 
questionnaires. One reason likely for this is that some widely used 
published instruments would not be applicable because of the unique nature 
and purposes of Innovations Program projects. Practical impediments to the 
use of published tests are that project directors and evaluatijnf consultants 
simply were unaware of the rang% of available tests, or that time constraints 
meant that it was not possible to wait until tests could be ordered and 
despatched from an interstate or overseas publisher. 

Searle and Parmenter's (1978) evaluation of a work preparation program 
for mildly intellectually handicapped children employed several published 
instruments to assess relevant student outcomes. Halpern, Raffeld, Irvin 
and Link's (1975) Social and Prevocational Information Battery (SPIB) 
supplemented school-developed measures of social competence and provided 
additional valuable information about students 1 knowledge of aspects of 
social and prevocational skills. The Burke and Sellin (1972) Self Concept 
of Ability as a Worker Scale was administered at the end of the program ? 
both to students involved in the project and a comparison group from a slow 
learners 1 class at a nearby high school. Payne, Mercer and Epsteins (1974) 
published personal/vocational checklist was also used to obtain ratings of 
the same studehts as expressed by, teachers, employers and the students 
themselves. The 25 items in this test are answered on a five-point ' scale 
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♦with the responses of Very Poor, Poor, Just Fair, Good and Very Good.. 
Examples of some of the items in this personal/vocational checklist are 
given below: 

. How well do you get along with other workers? 

Are you at work oft time? 

How clean and neat are you? 
. Do you think of safety when you are working? 
. Can you work without being supervised all the time? 

How well do you count? \ 

How well do you write? 

How well do you handle your money? 

Possibly the most interesting result emerging from the analysis of responses 
to this questionnaire was that opinions about students were mor» conservative 
among teachers and employers than among parents and the students themselves. 

Groundwater-Smith (1977) also used a published questionnaire in her 
evaluation of a project, involving several centres of learning. This question- 
naire was developed by Coppersmith and contains 58 items measuring student 
self-esteem* Each item is responded to by students on a two-poiJ?t scale 
with responses of Like Me and Unlike Me. Examples of some of the items in 
this published instrument are: 

I'm pretty sure of myself. 

I'm a lot of fun to be with. 
. I would rather play with children younger than me. 
. It's pretty tough to be me. 
. Things are all mixed up in my life. 

I'm not as nice looking as most* people. 

I'm a failure. 

\ < 
"Home-Made 11 Instrumeiafcs 

* Questionnaires developed especially for the purpose were used extensively 
in augmented evaluations as sources ol information. Wheteas some question- 
naires sought a fixed response (e.g., Yes/No) to specific questions, others 
provided space for open-ended answers to more general questions. Lehman's 
(1978.a) augmented evaluation of a module system of studies made use of 
very detailed questionnaires to obtain evaluative information from students, 
teachers, parents and the project director. For example, one of Lehman 1 s 
open-ended questions to parents was: "Please explain any differences you 
know that exist between the present Year 7/8 programs at Richmond River High 
School and the way these programs used to be." An example of a fixed-response 
question (Yes /No /Don 1 !: Know) on the student questionnaire was: "Are you 
getting enough time to study, to understand, and to complete each activity 
you uhdertake?" One of the open-ended questions included in Lehman's 
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teacher questionnaire was: "Please provide specific details of difficulties 
and problems you believe are evident when teaching your subject because of 
the module system," 

Howard's (1977) augmented evaluation of a science resource materials 
development project included an open-ended questionnaire which sought 
information from the teachers directly involved in materials development* 
Some of the items included in this questionnaire are listed below: 

Did your partic ipatioh cause any difficulties? 

What additional facilities or back-up services would you like in 
{ future similar projects? 

What skills did you acquire as a result of participation? 
What advice could you pass on to anyone wishing to participate in 
a similar project? 

What reactions in your colleagues at work did your participation cause? 

Johnstone's (1977) evaluation of a cyclical timetable project provides 
some colourful examples of how students' replies to open-ended questionnaire 
.terns can add interest to an evaluation report. -For instance, some of the 
replies received from Year 9jstudents in response to a questionnaire item 
[t the most preferred activities engaged in are listed below: 

'Cooking is best because you can eat the food you cook." 

'\eatherwotk is best because I finished the work., I started and it * . 
>oked like it was supposed to - not like my pottery did."' % 

"Kayaking is* best because you couLd swim and paddle around without / 
teachers saying to do this, do that, blah, blah, blah." 

Similarly, responses to another open-end'ed question dealing with student 
attitudes. to forced activities revealed several dissatisfactions: 

"Some of. the activities we have been made to do are totally boring 
unless you have* a .rather unique interest in them." 

"I've already got two Star bars in lifesaving and I'm just doing 
everything over and* over again." 

Likert Scales 

One of the mos't widely used approaches to. attitude measurement in 
educational research today is the method proposed many yeafs ago by Likert 
(1932). Table 4 shows an example of a Likert-type attitude questionnaire 
'Tised in an augmented evaluation study (Eraser and Koop, 1977). The project 
being evaluated involved the use of motivational materials related to the 
history of mathematics, and the questionnaire in Table 4\was used to measure 
the attitudes of a group of Year 9 students using a mathematical play (which 
was about the life of the ancient mathematician Thales), In "fact*,-* this 
questionnaire was 'administered as a pretest prior to studenfe studying tjie 
Thales play and again as a posttest after tSie completion o'f the play. 
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TABLE 4. An* Example of an Attitude Questionnaire with 
Likert-Type Items (from Fraser and Koop, 1977) 



lm'I like mathematics lessons more than any other 
lessons. '(+) 
* * 

2> It would be a waste of time learning about 
£he lives- of jnathematicians who lived 
thqusands of ■ years ago. (-) 

3,. I like to try to solve ^Jfemat ical 
puzzles and problems. (+) , 

4. I think mathematics is a dull and 
uninteresting subject. (-) 

,5, Outside the classroom, I don't like to 
* thit% abdtit mathematics. (-) 
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6. I like to make up sums and problems ^ 
' * #arf# do them- at home. (+) 

7. Learning about the history of mathematics 
- would be interesting. 



8. I don't like mathematics* because the examples 

are too hard and make me jthink. (-) 

* * * 

9. Mathematics is 6ne of the most* interesting 

subjects, and I want to do as much 

mathematics as possible. (+) r t 

10. Mathematics has few practical*applicat ions 
^ to daily life. (-) w , 

11. Mathematics is almost important subject and 

more time should be gfven to it at school. (+) 

*-* 

12. I dislijce -mathematics and will do as little * 

^mathematics as pbssible at school. 

13. I would enjoy rhathematics lessons more if they 

included some plays related to mathematics. (+) 

14. , Mathematics is a ^|ff icult .subject which I 

don't enjoy. (-) , 



SA A N D SD 



SA A N D SD 



SA A N D SD 



SA A N D SD 



SA A N D SD 



SA A N D SD 



SA A N D SD. 



SA A N D SD 



SA A N D SD 



SA A N D SD 



SA A N D S3^ 



SA A N D SD 



SA A N D SD 



SA A N D ■ SD 



Items designated (+) are scored 5, 4, 3, 2, 1 for the responses Strongly 
Agree (SA^JSAgree (A), Not* Sure (N) f * Disagree (D), Strongly Disagree (SD\. 
Items designated. (~) are scored in the reverse manner. Omitted or invalidly 
. answered items are given a neutral score of 3. 

As items. 2, 7, 10 and 13 measure specific attitudes, scares on these items 
cannoE be summed to form a meaningful total. Scores on the other 10 items can 
q be sunmied to form a general attitude to mathematics score. 
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It oan be seen from Table 4 that the essential feature of a Likert 
item. is that it. involves respondents in indicating their degree of agree- 
ment abodt various statements. In fact, the items in this questionnaire 
'have a. five-point response format, with the alternatives of Strongly 
Agree (SA), Agree (A), Not Sure (N), Disagree (D) and Strongly Disagree (SD). 
Item 1 (and otSier items designated "+") 4re scored by allotting five marks 
for the Strongly Agree response, four marks for the Agree response, 
and one mark for the Strongly Disagree response. Furthermore, in order to 
avoid response set, half of the items are worded in such a way that the 
reverse^scoring procedure is required (see items designated "-"). 

9 

A particularly noteworthy feature of the questionnaire in Table 4 is 

that attitude items fall into two distinct groups. The first group, 

namely Items 2, 7, 10 and^ 13, were written by the evaluators to. measure 

four specific pupil attitudes which could change during the time of use of 

the play. The four specific attitudes are attitude to learning about the 

lives of ancient mathematicians (Item 2), attitude to learning about the 

history of mathematics (Item 7), attitude to the J prac tical applications of 

mathematics in daily life (Item 10), and attitude to including mathematical 

plays in mathematics lessons (Item 13). Scores on these four attitude 

items were not combined to form a meaningful total score but, instead, 

were used as four distinct attitudinal criteria. The second group of 

attitude items comprise a 10-item attitude to mathematics scale developed 

by Keeves (197,4). A student's total score is obtained by summing scores 

obtained for Items 1, 3, 4, 5, 6, 8, 9, v ll, 12 and 14. K 

t~ — * 

The reasonfor using Keeves 1 attitude^scale in the evaluation was not 

k '*that it Was thought that pupils would undergo appreciable changes in general 

attitude to mathematics during the use of a single play. Rather, it was 

c r 

used to provide* information about whether the effectiveness 
of the play was different for pupils of differing attitudes towards 
mathematics. Recaus'e of the purposes to which the attitude to mathematics 
data were put,\a single measure of this attitude at one instance in time 
* would have sufficed. The attitude to mathematics items, however, were 
administered at pretesting and posttestiog for 'two reasons. Firstly, it 
proved convenient because it reduced the costs involved in printing two 
separate questionnaires and because it simplified teachers 1 instructions 
.for administration by having identical questionnaires on the two occasions. 
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Secondly, the intention of the four items measuring specific attitudes 
^relevant to the^^tLay was made a little less obvious to pupils by inter- 
spersing these^Lterais with the 10 items measuring general attitude to 
mathematics. 

When pretest and pasttest scores were compared for the four items 
measuring specific attitudes, it was found that statistically significant 
changes had occurred for 'Items 7 and 13 but not for. Items 2 and 10. 
That is, small and nonsignificant changes occurred during the use of the 

o 

Thales play in attitude to the statement that Lt voyuld 'be a waste of time 
learning about the lives of ancient mathematicians and to the statement that 
mathematics has few practical applications to daily life. On the other 
hand, a significant improvement in attitude was found towards the statement 
that learning about the history of mathematics would be interesting and 
towards using mathematical plays in mathematics lessons. 

Likert-type items were also used to measure student attitudes in 
augmented evaluations of-an integrated social science project (Lehman, 1978b) 
and an environmental education project (Blackburn, undated). In Lehman f s 
evaluation, students replied on .a five-point scale to statements such as 
"The study of geography is ... n (with responses ranging from Always 
Interesting to Never Interesting) and "Tolerating people who are different 
to me because they speak a different language is ..." (with responses 
ranging from Always Very Difficult to Never Very Difficult). In Blackburn's ' 
evaluation, students responded on a five-point scale (ranging from Very Much 
Likfc Me to Not At All Like Me) to self-concept questions such as "Finds 
native flora rather dr^b and uninteresting", "Likes to go exploring for 
aboriginal cave paintings and burial sites" and "Believes the dingo is a 0 
pest which should be shot out". Furthermore, by administering the same 
self-concept questionnaire both as a pretest prior to when students went on 
expeditions to the semi-wilderness and again as a posttest afterwards, 
Blackburn was able to detect numerous areas in which favourable and 
statistically significant attitude changes, had occurred. 

* 

Semantic Differential 

Another widely used approach to the measurement of attitudes is the 

semantic differential technique developed by Osgood, Suci and Tannenbaum (1957) 

Table 5 contains a simplified version of the semantic differential scale used " 

by Boylan (1977) in his augmented evaluation of .a project at Bournda Field 

Studies Centre. This table shows that students were asked to provide their 

attitudes towards the concept of "Field Studies Centre" by describing the 

* 
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concept in terms of several adjective pairs^(e.g; , good-bad). Each 
adjective pair is rated on a seven-point graphical sc^le by placing a 
tick in the appropriate location along , the line. 

TABLE 5. Semantic Differential Scale for Measuring Attitudes 
Related to a field Studies -Centre Project 
(from Boylan, 1977) 



The Field Studi'es Centre is:- 
good 

beautiful 
dishonest 
worthless 
necessary 
unimportant 



bad 

ugly 

hones t a 

valuable 3 

unnecessary 

important 



a t» 

For these items, a tick at the extreme left of the ^continuum is scored 1 
and a tick at the extreme right is scored 7. Other items are scored in 



the reverse manner, 
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The scoring of this semantic differential scale is similar in 
several ways to the scoring of the Likert scale discussed in the previous 
section.^ First, in-order to avoid response set r some items are given 
the maximum score of seven for placing a tick at the extreme left of the 
continuum while other items are given the maximum score for placing a 
tick at the extreme jright. Second, it is possible to consider scores on 
individual adjective pairs, or to sum the scores for all items in Table*5 
to obtain a meaningful total score* 

Boylan used a number of semantic differential scales to study ths 
effect on student attitudes of the experience of visiting th,e Bournda Field 
Studies Centre. In fact, a separate semantic differential scale was used' 
to assess student attitude towards the six concepts of "pollution", "the 
study of ecology", "ma^s use of the environment", "conservation", "the 
natural environment" and. 11 the Field Studies Centre". A group of 72 students 
from two high schools and one primary' school responded to the semantic 
differential scale both before and after visiting the Centre in order to 
provide measures of attitude change. Also a control group of 55 students 
(from one high school and on<e primary school) did not visit the Centre but 
still provided their responses to the semantic differential scales on one 
occasion in order to permit a comparison of the attitudes of non-visitors, to 
the Centre with the attitudes of visitors. 
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TABLE 6. Social Competence Checklist for Mildly Intellectually 

Handicapped Students (adapted from Searle and Parmetiter, 1978) 



General Area 



Specific Skill 



Teacher Check 



Knowledge of 
self, home ^ 
and family 



Knows and can tell name 
Knows address 

Can write telephone number 



Self-help 
skills 



Can tie shoes 

Can brush and comb hair 



Conmunicative "> 
skills 



Can use public telephone 
Calls adults by Mr $nd Mrs 
Can follow simple instructions 



Knowledge of 
school and local 
community 



Knows location of specific classrooms 
Can use train and bus timetables 
Can go shopping 



Safety > 



Caji cross roads safely 

Can recognise traffic and safety signs 

Can tell what to do if lost 



Occupational 
skills 



Is punctual 
Attends regularly 
Cooperates 



Knowledge of 
rights as an 
employee 



* , Knows about sick pay 

Knows about workers 1 compensation 
NjCnows about superannuation 
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A series, of significance tes£s revealed a number of statistically 
significant differences 'for the total evaluation scale score and, in each 
case, the direction r of these differences supported the efficacy of .the 
Field Studies Centre project. For example, evidence suggested that 
visiting ^students' attitude's towards mani^-use .of _ the . environment and 
towards conservation were more favourable '^than the attitudes of non- 
visiting students. 

f 

Checklists % * / 

A convenient way of obtaining information about student knowledge, 
"competence and skills is to use a checklist. Table 6 shows* an example of 
a simple checklist adapted from Searle and Parmenter 1 s» ( 1978) evaluation 
report for a work preparation program for mildly intellectually handicapped 
children* As the aims of this program include the development of social 
and vocational competence, the checklist shown^ in Table 1 could be filled 
in with j^icks by the teacher^ in order to provfde a_record of student 
progress towards developing competence in seven areas (namely, knowledge ' 
of self, home and family, self-help skills, cqmmunicative skills, knowledge 
of school and local community, safety, occupational skills, and knowledge 
of rights as an employee). 

* 

CONCLUSION 

■ 

This chapter discusses eight specific evaluation 41 techniques useful 
for enhancing the description of an innovation and fo£ providing an 
innovator with formative .evaluative information on which to base decisions 
about program improvement. The techniques considered are interviews, 
observation, student diaries, usage rates, n expert £e'%tinftny, 
discussions at meetings of project personnel, achievement tests, and 
questionnaires. Furthermore, it should be emphasise^ that usually it is 
desirable to employ several different techniques within the same evaluation. 
The advantages of using multiple evaluation techniques are that the 
information base is made larger and because the consistency of information 
obtained through use of different techniques may be checked^ 
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CHAPTER 4 s CONCLUSION 



The purpose of this final chapter is to provide further assistance 
to innovators in the task of evaluating and documenting their innovative 
efforts* This purpose is achieved in four ways. First, the material in 
previous chapters of the monograph is summarised and amplified. Second, 
it is suggested that further guidance can be- obtained by consulting some 
of the complete augmented evaluation reports which have formed the basis 
of the extracts reported throughout this monograph. Third, some of the 
theoretical evaluation: literature which might guide the evaluation activities 
^of innovators is briefly reviewed. Fourth, the potential merits of seeking : 
assistance from colleagues when conducting an evaluation are outlined* 

DISCUSSION OF PREVIOUS CHAPTERS 

The central purpose of Chapters 2 and 3 is to provide practical help 

to innovators attempting to evaluate and give an account of their school- 

level projects. In- fact, the primary audience for this monograph 

is present and prospective innovators, whether funded by the Innovations 

Program or not, who want guidance in evaluating and documenting their 

projects. Furthermore, although examples have been drawn throughout the 

monograph from reports of augmented evaluations of Innovations^rogram 

projects conducted with the assistance of -an external consultant^ the 

central thrust in this document is *to provide assistance for innovators. 

responsible for evaluating their own projects* Sec^n^ary "audiences which 

might al-so find the present monograph useful are Innovations Program * 

counsellors who couLd be asked by grantees to advise on project evaluation^ * 

and various preservdce and inservice educators interested in upgrading the 

practical skills of trainee or experienced teachers in the important task 

of evaluating educational programs, 

« 

In Chapter 2; Describing an Innovation, examples are given of the way 
in which other innovations have been described in terms of,. certain key 
features (e.g., aims, setting, material). These features of an innovation 
normally can be described by a project director even if no specific 
evaluation techniques have been used throughout the time of the project. 
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Also a description of these aspects of an innovation provides a very useful 
basis for the dissemination of ideas to others who might- be interested in 
attempting a similar innovation. Chapter 3: Evaluation Techniques 
represents an^extension to Chapter 2 because it provides illustrations of 
specific evaluation techniques (e.g., interviews, questionnaires) which can 
be used throughout an innovation for two reasorfs. First, the description 
of an innovation can be enhanced considerably if it includes evaluative 
information collected during the life of the innovation. Second, by 
collecting information via these techniques during an innovation's life, an 
innovator is provided with formative evaluative information which can be ^ 
used to guide important decisions about project revision. 

Table 7 is a summary of the major headings contained in Chapters 2 and 3. 
That is, this table provides a checklist of seven key features (namely, aims 
and rationale, setting, problems) which could be considered when 

describing an innovation and of eight potentially useful evaluation 
techniques ( interviews , observation, questionnaires) which might be 

used to advantage to collect evaluative information during a project. 

It should be emphasised, however, that Table 7 does not provide a 
"recipe" for documenting and evaluating any innovation. As the Schools 
Commission (1975b, p. 4) correctly notes, "the kinds of information which 
are most useful will vary according to tjtie nature and aims of the project. " 
Consequent ly,. Table 7 should be considered simply as an array from which an 
innovator might choose several features for describing a project and several 
evaluation techniques suitable for the particular project being considered. 
The reason for assembling the material in Table 7 as a convenient checklist 
is so that an innovator might profitably refer to it at regular intervals 
throughout the life of a project. 

Although epch evaluation technique is discussed separately in Chapter 3, 
it is usually desirable to employ more than one technique in any particular 
evaluation. This combining of evaluation techniques can provide a broader 
range of evaluative information, enable checking of the consistency of data 
from different sources, and help to reduce evaluator bias. Furthermore, 
although curriculum evaluation writers are divided about whether quantitative 
or qualitative evaluative information is preferable, there can be real 
-advantages in combining some of the quantitative approaches (e.g., questionnaires) 
and qualitative approaches (e.g., student diaries) described in Chapter 3 
in order to provide a more useful information base (see Smith and Fraser, 
in press) . 
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TABLE 7. • A Checklist -of Features for Describing an Innovation an* 
'Evaluation Techniques 



FEATURES FOR DESCRIBING AN INNOVATION 

- r 

• AIMS AND RATIONALE , . 

• SETTING 

• ORIGINS OF PROJECT ' 

* • MATERIALS AND EQUIPMENT 

• STUDENT ACTIVITIES ~~ 

• FUNDING 

• PROBLEMS / *, 

-Feelings of isolation 

- Time delays 

^Staff mobility / 
„ * -.Failure to establish need for innovation 

- Ineffective communication between 

innovators and others 
t - 

EVALUATION TECHNIQUES 

• INTERVIEWS 

A 

• OBSERVATION 

• STUDENT DIARIES I 

• USAGE RATES * 

• EXPERT TESTIMONY 

• DISCUSSIONS AT MEETINGS OF PROJECT PERSONNEL 

• ACHIEVEMENT ' TESTS 

• QUESTIONNAIRES 

- Published instruments 

- "Home-made" instruments 

- Likert scales 
-Semantic differential 
-Checklists 
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Just* as it is important m to use a variety of techniques for obtaining 
evaluative information, so too is it critical in an evaluation to seek 
.the opinions of a variety of different people (e.g., project personnel, 
teaching colleagues, the school principal, students, parents). In fact, 
Hardy's (1979) augmented evaluation of a community counsellor project 
revealed that, among various people consulted, "there was a marked 
divergence of interpretation of what had 6ccurred and what was occurring" 
(p. 99). It is for this reason that Stake (1976) has recommended that 
differing value perspectives present be referred to when reporting the 
evaluation of atiy project. 

FURTHER GUIDANCE FROM AUGMENTED EVALUATION REPORTS 

Chapters 2 and 3 of this report have drawn heavily on material con- 
tained in augmented evaluation reports to illustrate how to describe 
innovative projects and to suggest some useful techniques which can be 
employed in evaluating innovations. Although the examples contained in 
these chapters are necessarily brief, the reader is urged to seek fuller 
details by consulting. some of the complete augmented evaluation reports, 
which are obtainable from the Education Information Retrieval System 
(E.l.R.S.).* 

As well as providing amplification of issues discussed in Chapters 2 
and 3, the complete augmented evaluation reports can also be referred to 
for clarification of many other important issues and problems which can 
arise when conducting an evaluation. For example, complete evaluation 
reports^provide guidance in ways of analysing evaluative information and 
ways of organising and presenting an evaluation report. For example, Jones*s 
(undated) augmented evaluation report of a remedial mathematics program 
illustrates how a variety of information can be brought together in an 
interesting and revealing way as student case studies- These case studies 
for individual students involved a trtJT-page description based on the 
results of diagnostic tests and interviews conducted with the student and 
the class teacher. Different augmented evaluation reports also provide 
examples of the* use of different writing styles (e.g., academic, colloquial) 
in reporting an evaluation, and illustrate the use of tables, diagrams or 
photographs in making communication more effective. 



The address of E.I*R.S. is Wattle Park Teachers Centre, 424 Kensington Road, 
Wattle Park, South Australia 5066. 
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' EVALUATION LITERATURE FOR BUSY INNOVATORS " 

A major strength of the present monograph is that it provides practical 
y assistance with evaluation by concentrating on concrete examples of useful 
approaches and techniques followed in the evaluation of other recent school- 
level projects. Although this very practical orientation is laudatory," 
further guidance in the task of evaluating innovations can be found in the 
host of potentially useful evaluation ideas and models contained in the 
theoretical literature of curriculum evaluation. Consequently, those 
interested in conducting and reporting a thorough evaluation of their 
innovations might wish to use this practically-oriented monograph in 
conjunction with more theoretically-oriented literature. 

Because the gr^at majority of -innovators are busy people with little 
knowledge and experience in evaluation, they are likely to need assistance 
if they are to make effective use of literature in evaluating their projects. 
This is especially 'true because the currdcuAjytm evaluation literature is 
voluminous and widely scattered, and because it-reflects great diversity of 
opinion-about desirable evaluation approaches. Moreover, because of the 
idiosyncratic nature of each' innovation, there can be no evaluation ,f recipe M 
which would be appropriate for all projects. Consequently, what is needed 
by busy innovators is a relatively brief overview of a wide range of 
evaluation ideas and methods, so that innovators can identify literature 
pertinent to their own situations and purposes and follow it up in more 
detail, ' In fact, an annotated bibliography of curriculum evaluation 
literature (Fraser and Houghton, 1982), which provides a one-page abstract 
of over 100 'curriculum evaluation references, currently is available ^ 
to provide busy people with a brief overview of a wide range of theoretical . 
evaluation literature. 

An attempt to identify and review a representative sample of references 
fijom the theoretical curriculum evaluation literature is outside the scope 
and purposes of the present monograph. Nevertheless, it may be of assistance 
to readers to' list here a handful of potentially useful references. Straton 
(1975) and Wines et al (1975) have written papers specifically on the 
evaluation of innovations, while Steadman (1976) offers an informative 
chapter on techniques of evaluation. Owens and Evans (1977), Fensham^sL^&O) 
and Groundwater-Smith and Nicoll ( 1980b) ~have published straightforward 
booklets for those wishing to improve their evaluation skills, while The 
Curriculum Development Centre has provided a monograph which gives a good 
overview of the major theoretical writings and ideas in the field of 
curriculum evaluation. Among the more useful general textbooks devoted to 
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evaluatipn are Taylor and Cowley (1972), Weiss (1972), Worthen and Sanders 
(1973), Stake^'1975), Stenhouse (1975, ch. 8), Hamilton et al (1977), 
Anderson and* tall (1978) and Cuba (1979) . Also, the Teachers as 
Evaluators Project (1978-80) has produced a four-part bibliography of 
materials and references on program evaluation for teachers, 

Scheyer has proposed ..a relatively simple approach which innovators 
might apply in collecting their own evaluative information on their own 
projeqts (Scheyer, 1975-76; Scheyer and Stake (1976). This approach • < 

involves people keeping self-evaluation portfolios which are described ^ 
in the following way: , % * 

The idea is to establish a file or collection of recor-ds and 
materials which broadly represent the program. This portfolio 
should be a loose collection so that parts of it can be 
rearranged and differently displayed from time to time. Th£ 
entries should reflect the program activities, its issues, its 
valuings, and its compromises. (Scheyer and Stake, 1976, p.* 37) 

Things which might be included rn a portfolio are handouts, minutes of 
meetings, photographs, audiotapes, examples of children's work, records 
of observations, anecdotes, verbatim quotations, transcripts of dialogues, 
etc. In particular, a self-evaluation portfolio should reflect the valuings, 
judgements and opinions of various persons involved in the program because 
diverse testimonies on important evaluation issues reveal the different 
perceptions that people have about what the strengths and weaknesses of 
the program are. Furthermore, the experience of portfolio keeping helps 
to train project staff in a self-evaluation strategy which endures beyond 
the lifetime_of the project. 

A number of augmented evaluation reports can be referred to for 
clarification of the way that theoretical evaluation literature might be 
employed to advantage in conceptualizing, planning "or reporting the 
evaluation of an innovation. In particular, the paragraphs below describe 
the use of theoretical evaluation models in augmented evaluation reports, 
including Ingle's (197,8) use of Stake's (1967) countenance model and 
Searle and Parmenter's (1978) application of a systems evaluation model 
based on the ideas of Schulberg and Baker (1968), Budde (1972) and 
Mitchell (1975). 

Stake's (1967) countenance model is composed of two dimensions. The 
first' dimension consists of antecedents (which embrace goals and materials), 
transactions (which involve student activities) and outcomes (both student" 
outcomes and- outcomes on other people). The second 'dimension - 'which 
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consists of Intents , observations , standards and judgements - highlights 
Stake's claim that *there is more to the evaluator f s task than recording 
observations -of a variety of relevant aspects related, to antecedent^ 
transactions and outcomes. As well the evaluator should consider the 
intents (e.g., planned-for environmental conditions and student 
achievements), the standards or ratings considered meritorious by 
different" reference groups, and judgements *(i .e. , the evaluator should - 
record his judgements by assigning weights to each set of standards). 

Ingle's (1978) augmented evaluation report ior"TuT^^Q[nunity Reading 
Centre" project used Stake's countenance model atj the planning stage to 
ensure that antecedents, transactions and outcomes were adequately 
represented in the information collection plan. Table 8 shows the , 
information sources contained in Ingle's evaluation plan and classifies 
these i nfor mation sources into antecedents, transactions and outcomes. 
This table shows that examples of antecedents include physical plant and 
teacher characteristics, examples of transactions include Ume allocation 
and sequence of events, and examples of \utc9mes (include student achieve- 
ment and attitudes and effects on teachers". 



TABLE 8. Information Sources for Evaluation 6f a Community 
Reading Centre Project Classified afe Antecedents, 
Transactions or Outcomes (from Inglte, 1978) 



Stake's (1967) 
Category 



Information Source 



Antecedents 



Student characteristics 
Teacher characteristics 
Curricular content 
Curricular context 
Instructional! materials 
Physical plant 
School organization 
Community context 



Transactions 



Communication flow 
*Time allocation 
Sequence off events 
Reinforcement schedule 
Social climate 



Outcomes 



ERLC 



Student* Achievement 
Student Attitudes 
Student potor skills 
Effects /on teachers 
Institutional effects 
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Figure 5 shows how a systems evaluation model was applied itv-the 
evaluation of an innovative work experience project for mildly/' 
intellectually handicapped children (see Searle and Parmenter/ 1978: 
Parmenter and Fraser, 1980). This figure shows that the essential 
elements of the y sys£em§ approach are inputs , which usually lead to a 
statement of a problem, process , which is an attempt at solutions to- a 
- problem, and odtputs , which are the results obtained. The modSl also 
i^LQ^rgora^^ an information feedback loop which may lead to program 
adaptation* 

The main reason for adopting a systems approach in this . evaluation 
was so that, in addition to a documentation of the, final outcomes, 
there also was an analysis of the total dynamics of "the program. Thus 
it was hoped that other schools' might be able to get a clearer picture 
. of all aspects of the organisation and execution of the work experience 
- ri program. One of the essential strengths of a syste^p approach is that 
> it gives an opportunity for both ori-going or process evaluation and 
output or product evaluation to be conducted. Consequently an 
organisation conducting a similar program might find answers to 
<S questions about whether the goal's of the program are being achieved, 
whether there is an effective coordination of the organisational sub- 
units, whether the necessary resources are being acquired and 
fc maintained, and whether* the organisation is adapting to its environment 
and its own internal demands. 



Figure 5 shows that the inputs for the evaluation included the 
population served and its needs, together with the situational variables 
of qommunity, school, economic climate and parents. Population 
characteristics included in the eval-uatipn were I.Q., vocational £est 
results and recommendations, academic and social attainments, and 
teacher ratings of social and emotional adjustment. The school's 
characteristics and its value system were described, particularly the 
baseline conditions which operated prior to the implementation of the 
innovative program. Also the evaluation report listed the corporate 
goals and attitudes held towards the students by various sections of * 
the community '(e.g., employers, government agencies, regional offices 
of education). 
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L. 
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Program 
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OUTPUTS 



Intermediate 
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FIGURE 5. > Systems Model for Evaluation^ a Work 'Experience Program (taken from Parmente 
and used in augmented evaluation conducted by Searle and Parmenter, 1978) 



r and Roberts > 1979, 
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Whereas inputs in the model in Figute 5 may be seen as the statement 
of. ar problem to be solve?) (i.e., given this population and its vocational 
and social .needs, how may we h»est meet them?), £he process should be 
attempts at a solution. Here were stated (and quantified wherever 
appropriate and" possible) the program's objectives, the strategies 
adopted, the resources made available, and the dynamics of the "system. 

if 

These are penary independent variables that can be subject to experimental . 
management and hence are crucial to feasibility and transportability 
decisions for similar programs in other settings. Objectives were stated 
wherever practicable in operational terms of what a student would le^rn , 
in the various areas of activity (prevocational , vocational and technical).' 
Techniques and strategies employed were identified' (rf.g.j teaching 
procedures, special programs, group organization)-. Resources were 
identified including additional staff facilities (e.g., technical education), 
community involvement , and equipment. Also, the dynamics of th£ organization 
of the process variables were monitored and described. Of particular 
relevance was the response of the organization to feedback from the ongoing 
evaluation of outcomes. 

Figure 5 shows that the outputs considered in the evaluation may be 
divided into intermediate and long term, and that altogether five maip areas 
of outputs were studied (specific skills, social competence, self concept, 
independent living and job placement). The first area of outputs considered 
was specific skill development. Teachers were encouraged to develop on-going 
assessments for each of the students, and particular attention was paid to 
the success or otherwise of various sections of the work. Social competence, 
a6 measured by school-developed and published instruments, provided a second 
area of outputs for study. The third outputs area studied was self develop- 
ment as measured by the 'Burke and Sellin ( 1972) Self Concept of Ability as a 
Worker Scale, 

ASSISTANCE WITH EVALUATION FROM OTHERS V 

The Schools Commission's ( 1975bJ||guidelines for evaluation urged \ 
innovators to enlist the assistance of a colleague or friend when planning 
and conducting an evaluation. Particular merits .of seeking assistance from 
others are that project directors coald be too close to the day-to-day 
conduct of a^roject to take £n objective view or to obtain frank opinions 
from others involved. Nevertheless, despite, the potential benefits of 

enlisting the assistance of others, Edwards 1 (1980) review of final reports 
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of projects funded by the Innovations .Program in N.S.W. reveals that only a 
very small fraction of grantees had sought any assistance from others. 
Despite this, Edwards 1 questionnaire survey of a sample of y grantjees 
indicated that the type of help with evaluation most wanted was face-to- 
face c<?ntact with people wi£h experience in evaluation. 

In addition to the guidance with practical aspects afforded by the 
present report, innovators might gain additional help with the evaluation - 
of their project by enlisting the kelp of others. People from whom 
assistance could be sought include teaching colleagues, people from other 
schools interested in similar innovations, or people from universities 
and colleges with experience in program evaluation. For those innovators 
funded by the Innovations Program, ijt also is possible to'enlist the- , 
assistance of Program counsellors, some of whom have particular experience 
in the area of project evaluation* 

Augmented evaluation reports provide good-examples of the way that 
innovators worked in close liaison with external people experienced in 
evaluation. Moreover, these augmented evaluation reports suggest some 
tentative guidelines for ways of optimizing the effectiveness and usefulness 
of assistance with evaluation from others. For example, the fact that the 
effectiveness of some evaluations (e.g., Boylan, 1977; Howard,. 1977; Ingle, 
1978) was impeded by the large distance between the innovator's location 
and the location of the person assisting with evaluation suggests the 
desirability of innovators seeking help with evaluation from people located 
at a small enough distance to fillow regular face-to-face contact. Cither 
augmented evaluation reports (e.g., Lehman, 1978b; Searle and Parmenter, 1978), 
by revealing that important evaluative information can be missed if assistance 
with evaluation comes too late in the life 6f a project, underlines the 
importance of seeking help with evaluation near the commencement of a project. 
Casein's (1977) finding that students were unwilling to make negative 1 
comments about an innovation to project personnel reinforces the notion 
that ^enlisting the help of external people could result in the acquisition 
of more frank opinions about a project. 
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DISCUSSION 

v 

In his discussion of the evaluation of innovations, Straton (1975) 
makes the following claim: 

Where evaluation studies of innovative educational programmes 
are planned, there is a need for sensitivity to the particular 
project and its setting, ... a need for flexibility and 
originality in the procedures for data collection, a need for * 
a balanced description taking account of all types of infor- 
mation and a need to accommodate the rights of all audiences 
of the programme. Given the fulfilment of these needs, 
evaluation studies have an inmense potential for helping to 
improve the quality of Australian education by providing 
information about innovative educational programmes which 
is useful for fully describing the programme apd for making 
informed judgments and decisions. (Straton, 1975, p. 14) 

The primary purpose of* the present monograph is to assist innovators in 
their efforts in the important task of documenting and evaluating their 
innovations. To this end, Chapter, 2 identifies key features for describ- 
ing an innovation, Chapter 3 dlseusses evaluation techniques which can be 
used to enhance project documentation and to provide ongoing feedback 
which can lead to project improvement, and the present chapter suggests 
how further guidance might be obtained either from the theoretical 
evaluation literature or by seeking the assistapce of other people. 
It is hoped that, through innovators* use of this monograph, the 
evaluation and reporting of innovations might be enhanced so that, 
ultimately, the improvements in the quality of Australian education 
sugges&4d by Straton might be realised. 
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